I've always believed time travel was possible (I think we're travelling through time right now), but I don't think it would work exactly the way it did in the movies.
First off, travelling through time is travel through only one dimension, leaving yourself stationary in reference to the three primal dimensions. Imagine stepping into a time machine, and then stepping out into the cold vacuum of space because the Earth has continued its orbit while you were busy getting your autograph book ready for Jesus. Even if you tried to synchronize your flight plans with the revolution of our planet, you would be 30km/s closer to understanding Newton's Laws of Motion.
Speaking about physics (from an untalented physicist's point of view), shouldn't time be considered as flowing in more than one direction? It feels pretty unbalanced from a physical standpoint, to have time flow in one direction but not (the/an) other. Also, when I think of time flowing in one direction, I think of a ray, with a definite starting point, shooting off toward infinity. If time were infinite, however, wouldn't it go in both directions like a line? Or in every direction? A scenario like this just feels like it clears up the idea of moving backwards through time. Of course, I've always been under the impression that time was more or less infinite, though I don't really have the cognitive capacity to understand infinity as anything more than a circle. Maybe somebody could help clear that up for me.
If I'm right, however, and time is interminable and time travel is possible, the reason why there weren't a googol people at the birth of Christ or Elvis's first concert is because people will have killed themselves off before the secrets of chronogation are made apparent, or they became truly enlightened and had a good, working understanding of how time can be utilized, thus realizing the true havoc what could be wreaked should the timestream be tampered with. With this kind of knowledge, the Civilization would either actively bar lesser beings from time travel (which they may already have done), or work to reconstruct our timeline (you know, like in Peabody's Improbable History or Time Squad) until it was silky and smooth and unable to be fucked-with by some kid with a bicycle and an overdose of riboflavin.
So, in closing, I think time travel will be possible sometime after the Singularity, but by that juncture we won't need it or will see it as a burden.