I wouldn't say I always think about it when going to the toilet, but everytime when I see those hygiene-promoting ads, or somebody walking out of the toilets without washing her hands, I ask myself: " Why should I wash my hands??" It doesn't make much sense.

Some lab experiments during my course Introduction Microbiology (now 8 years ago, but it sure did make an impression). One experiment was to simulate the toilet-going-ritual:
1. petri dishes with a bacteria cocktail containing E. coli and some cocci et. al.
2. Touch it with your bare hand, 2, 4, 6 and 8 layers of toilet paper (we used the grey recycled type).
3. Then leave your hands unwashed, rinse with water or wash them with water and soap.
4. Last parameter is yes/no using the faucet.

There were stunning results in all groups doing these tests: even after using 6 layers and washing your hands, those damn bacteria were hitch hiking on your body. Only after 8 layers of paper + washing the culturable bacteria were negligible.
And who knows about that? Who is going to count 8 layers of toilet paper every time?
Even if you do, there's another problem: the water tap. You're "safe" if it's one with a handle, push button or an automatic one. But think about it the next time, do you turn the faucet open with your dirty hands, leaving some bacteria behind on the metal? You know, after washing, you close it with your clean hand and you got your bacteria back.

Last step is the door handle. If there's one person before you who didn't wash his/her hands, you still can't escape those "toilet bacteria". Same problem as with the faucet.

There's an option though if the above doesn't give you a comfortable idea; something I was told during my tropical hygiene class: use only your left hand during your toilet activities so your right hand will stay relatively clean for eating and shaking hands.

But regarding those bacteria, you can't escape them, You are never alone.