Speciation is defined as the origin of a new species, and the generally accepted definition of a species is defined as a set of individuals who can produce viable, fertile offspring with and only with one another. Donkeys and horses are closely related enough to be able to produce mules, which are unfertile but viable.

Apparently, Humans, Klingons, and Vulcans are at least as closely related as donkeys and horses, because there are instances in Star Trek of viable crosses between both a Human and a Klingon (B'Elanna Torres) and between a Vulcan and a Human (the ever-joyful Mr. Spock). Torres has had a daughter with Tom Paris, a human; I am unsure whether Spock (or other hybrid characters) has ever produced offspring, but the very fact that he is viable suggests a very close genetic relationship between these species, even though they're from different planets and all.

Any one know how the hell this works?

Update, a few minutes later: Apparently, Oslo knows. Here's what he says:
Oslo says re Speciation in Star Trek: I believe a two-part episode in TNG explains this uncanny convergence between species of different planetary origin; A single, super-evolved, highly-intelligent species evolved millions of years in advance of TNG. This species, finding itself alone in the universe, spread its genetic material among several planets, in pieces which could be discovered, combined, and decoded, revealing a message from them to the species who collected the pieces. The message, as I recall, was one of peace, which was ironic since the Klingons thought it was a secret weapon, the Romulans some other kind of secret, etc.
Thanks, Oslo.

Other notes on speciation in Star Trek (thanks m_turner):
The Romulans and the Vulcans split "recently", i.e. after the Vulcans got warp technology. Whether or not this can be called a speciation I don't know, since they can not only produce fertile, viable offspring, but there hasn't been enough time in evolutionary terms to cause any genetic shift. If eventually there is speciation, it will be a classic example of cladogenesis, however. Also, there is, somewhere in the canon, a Klingon-Romulan with whom Worf fell in love, another inter-species hybrid.