Time travel necessitates translocation along all axes.
Let's say we have a time machine akin to Doc Brown's DeLorean or The Doctor's TARDIS. Out of these, only one has a multidimensional translocation system, while the other has a simpler temporal translocation system. My grief with the latter is the intuitive reasoning that your time machine, having travelled X units of time along the fourth dimension, naturally must have travelled the appropriate amount of units along all the other dimensions, to end up near where you began. Let's make an example.
You have your DeLorean (the flying-equipped one with Mr. Fusion) and the year is 2014. Say you'd like to travel back to 2000 and instigate the Y2K bug for personal reasons. And say that time travel has been proved technically possible, so you're not too afraid of burning up or causing a gravitational singularity.
Exactly how do you return to the same physical coordinates that Earth had at that point in time? Is gravity so strong that it would keep your position pretty much the same? If not, my rambling continues.
Bear with me, I'm a jolly amateur. We all know Earth rotates around an axis in a barycentre placed somewhere between the centre of the planet and the moon, right? Right. We also know Earth orbits Sol, and the moon orbits Earth and Sol, right?
Well. The sun also moves through space as the Galaxy revolves around its axis. In addition to being affected by all the other objects in the Solar system and Galaxy, pulling and pushing and gnawing and mushing each other up. As well, our scientists have postulated that there is Dark Matter, and a mind-numbingly high number of other galaxies and giants strewn around the local supercluster, making the calculations needed to find the correct coordinates nigh impossible without first mapping every single massive object capable of affecting your starting position relative to all other positions, AND simulating the universe planet for planet until you reach the desired time. That is a lot of simulation for a computer bug.
If you were indeed to travel, say three hours, back in time with your DeLorean, you'd pop out 13 million kilometres away from your starting point. if not further out. As a result, you'd die of radiation and vacuum boiling, the deciding factor being if you at all end up near Sol. Isn't time travel fun, guys?
However, with the (slightly magical) TARDIS, destination time and place are parts of the same calculation, and the Time Lords do know their math machines.
My point is this: we can and have and will argue about the technical and romantical plausibilities of time-and-space-travel for as long as we possibly can, but it seems that even in a universe where spatial momentum is preserved along all other dimensions, you're pretty fucked if your time machine doesn't have a long range teleporter built in. And if we've developed a long range teleporter, why would we want to travel in time? We've finally unlocked the universe, let's turn off our computers and past-history-anxiety and go forth to conquer the Galaxy with pyramids and monoliths, to fuck with future archaeologists.
Amirite?
The hordes of time travellers aren't missing, they just found something better to do with their technology than fucking with yours. Or they're somewhere out in the black, regretting their hasty conclusions to impossibly complex problems.
TL;DR: The technology necessary to make time travel possible also makes the wielders into demigods. When you can move stars around for energy utilisation, why would you want to mess with past history?