According to the Jargon File (see Mongolian hordes technique), the term originally referred to an orgy! However, once the military began using a similar phrase to refer to situations ranging from a logistical SNAFU to a bloodbath (see clusterfuck), I can see the changeover described by sl0throp occuring.

Why "Mongolian?" If we look at the associated jargon file definition, then it probably refers to the idea of Genghis Khan's ancient Mongol hordes, and their bringing of terror and disorganization. A distant second-place guess would be some relation to the term mongoloid, which was at one point an acceptable descriptor for a person with Down's Syndrome (it has since been abandoned, because of its inaccuracy and its unfairness to people of mongoloid races).