This is a phrase that is good to utter when people say silly things. Here are some examples:

Me: I ate an elephant sandwich for lunch
Greenpeace person: You have to be kidding.

Me: I had a threesome with Winona Ryder and Famke Janssen last night.
Person who actually gets sex: You have to be kidding.

Me: I'd like to buy a Death Star please. Oh hell, make that two, one for me and one for the little lady.
Sienar Fleet Systems guy: You have to be kidding.

You can also say this when people say things that are serious and true but you can't bring yourself to believe them:

Me: George W. Bush is the US president.
Everyone I told: You have to be kidding.

Me: I had a threesome with Winona Ryder and Famke Janssen last night.
Person who actually gets sex:You have to be kidding.
Me: No, it's quite true, they are cooking me breakfast right now. At least that's what they said they were doing.... Judging from the moans of pleasure coming from the kitchen they are having wild lesbian sex. I better go help them.... *click*

What? It could happen!