There are certainly racial distinctions in Tolkien's work. How often do we hear the stout men of Gondor curse the Southrons or Easterlings?
But the difference is that these peoples have been corrupted by influences outside of their control. Men other then the three houses of the Edain were ensnared by the empty promises of Morgoth. The Southrons, especially the Corsairs, fell first under the sway of the Black Númenoreans (not a color, BTW) and later directly under Sauron's thumb. The Easterlings, primarily the men of Nurn and Rhûn never had a chance, so close were they to Mordor. The Edain were just lucky, or spared from treachery.
It should also be noted that the Númenoreans eventually returned to Middle-Earth as conquerors themselves. And what about Ar-Pharazôn the Golden himself, humbler of Sauron? It only took a few years for him to listen to the hostage Sauron's whispers about the Undying Lands, and so the fate of the King's Men and Númenor itself was sealed.
Finally, you could argue that the waning of the Dúnedain was due to a "mingling" of pure and unpure blood. Tolkien himself, in Appendix A to LotR, states that this waning was due specifically to the removal of the Men of the West from close proximity to Aman.