The phrase "mare's nest" has two distinct meanings. It originally entered into English through a translation of Giovanni della Casa's 1558 work Il Galateo overo de’ costumi; the English edition, translated by Robert Peterson in 1576, contained the line "Nor Stare in a mans face, as if he had spied a mares nest."
In this case, and for the next 250 years, a mare's nest was an illusory discovery -- if you found a mare's nest, you thought you had found something amazing, but you were sadly mistaken. There was also a distinct overtone that what you had found was patently ridiculous, even to the discoverer. For example, it was used this way in Jonathan Swift's play Miscellanies (1751): "What! Have you found a mare's nest, and laugh at the eggs?" This connection of humor strengthened throughout the years, and in the 1811 edition of Captain Grose’s Dictionary of the Vulgar Tongue mare's nest is defined thus: "He has found a mare's nest, and is laughing at the eggs; said of one who laughs without any apparent cause."
By the early 1900s a second meaning was emerging; a mare's nest now referred to a messy location or confused situation, perhaps in confabulation with the phrase rat's nest. This definition is now by far the more common of the two, and while this is hardly a common idiom, it is still in use in many parts of the world.