According to the bible, when a man and woman have sex, they become as one. And that's as good as marriage in God's eyes, apparently.
This is why a couple (at least by classical Jewish standards) wasn't actually considered married, despite the ceremony, until they had consumated the thing on the wedding night.
(It would probably be an interesting side point to figure out when things switched, and the ceremony became far more important than the sex. Probably about the same time they started sending out Bridal catalogues.)
The modern view that one should only have one wife actually comes from the New Testament, via Paul. Paul is listing, at one point, the proper elements necessary for one to be considered an elder of the church, and one of those factors is to be the faithful husband of one wife.
This is significant because in most churches, it's considered the goal of most people (or at least, by a traditional sense, the men) to eventually be considerable as elders, deacons, or what have you, and thus they'd all want to follow those rules. Thus why having only one spouse is "the thing".
The other, more cultural factor behind this is that much as Christianity came from Jewish sources, the church itself came out of an era of Roman/Grecian cultural influence, which generally held with only having one wife at a time. These two factors basically guarantee against polygamy.
Edit 7/31/2008: Talk about going off-course.
One of the prime themes of the bible is that we are created to be Relational. We need other people, we need relationships. Especially with God.
Genesis is full of this...God says "Let us make man in our image..." God walks through the Garden with Adam, God says that it isn't good for Adam to be alone, etc.
And so much of sin, intrinsically, is about things that break relationships and community, and keep us from being close to each other, and trusting each other.
Sex, culturally, has come to be understood as this really fun, arousing, adventurous thing two people can do on a Friday night when there's nothing better on television, and the new leather restraints have just come in the mail. It's an activity unto itself, within only itself as it's own horizon. This is, in fact, out of context.
Sex, contextually, is an outpouring of a relationship between two people, the point where they can't really say how they feel about each other in words any more, and they have to retreat into sheer physical contact.
When that relationship isn't there, when there's no commitment, no history, we run into two problems. First, the question becomes "okay, we've done all of the average positions in this book, and we're bored, this isn't working, what's next?" Because you're having sex to have sex, not because you've got this intensity from your relationship. And second...when you leave each other after the sex, you break something that was building between you, this added intimacy within the relationship built through the sex.
And you can only break that so many times before it can't be fixed again.