It depends on whether it is the subject or object. The distinction can be seen with these
useful examples:
It is unlikely that you'll ever meet anyone who really cares whether or not you ever say "whom" again.
It is unlikely that you'll ever meet anyone to whom it matters whether or not you ever say "whom" again.