Before discussing the intended topic, the reader needs to understand the idea of the Hubble radius. This explanation will remain non-technical, but necessary to comprehend the "real" topic.

The Hubble radius describes the current size of the universe, based on the (reasonable) assumption that light from a point-source origin of the universe (ie, the big bang) travelled radially outward from that point.

Simply put, anything the big bang threw out can only have travelled, at the speed of light, a certain finite distance. Our universe, therefore, consists of a perfect sphere having a radius the exact distance that light can travel since the big bang, roughly 12 billion light-years. No more (which would require something to travel faster than the speed of light), and no less (since the big bang did emit quite a lot of energy in the form of light, that light can only have gone radially outward at the speed of light).

This idea of a forced radius of the universe due to light travelling outward works both ways, however. Considering it from the opposite direction gives rise to the idea of the causal horizon.

Most people realize that light from distant objects takes some time to reach us. The light we see from our sun takes just over 8 minutes to reach us. This means we actually "see" the sun as it existed 8 minutes ago, not as it stands "now". As we get further away, this seeing-into-the-past effect grows larger - Sirius, the brightest nighttime star in our sky, lies 8.7 light-years away (fairly close, for a star), meaning that we see it as it existed 8.7 years ago. Quite a lot further away, we can see quasars near the edge (as defined by the Hubble radius assuming a universe roughly 12 billion years old, of course) of our universe, roughly 10 billion light-years away.

Now, imagine the existance of something that did not result from the big bang. Given the speed of light as the fastest rate at which information can travel through space, in order for us to detect such an object, a certain amount of time must have passed, just like it must for the sun, Sirius, and those quasars. But consider the quasars - At their distance, it took 10 billion years for their light to reach us. Something even further from us would take a proportionately longer time before we could see it. What happens when it would take longer than the age of the universe for light from a distant object to reach us?

Thus we encounter the essence of the causal horizon. Light from an object 20 billion light-years away could not yet have reached us. This doesn't just state that we can't see it - In any meaningful way, such an object simply does not exist in our universe.

Update to rebut Briglass' argument: I wrote the above explanation from the purely classical perspective. I didn't mention such ideas as inflation, or the possibility of an anisotropic universe, or the EPR paradox of quantum physics, or Robertson-Walker, or even (most of) relativity (other than the idea that nothing can travel faster than the speed of light), because I wrote it for the average reader, rather than for physicists and cosmologists.

I will grant that perhaps the word "perfect" goes a tad too far, but even taking every possible variation on the speed of light (short of it not existing as an upper limit in our universe), the geometry of the universe, and the flatness (or lack thereof) of spacetime into consideration, the core idea remains sound and the actual distances involved would change very little (percentage-wise).