They originate from breaking the letter
eta,
H, into two halves, and were devised by
Aristophanes of Byzantium, librarian at
Alexandria, in the second century BCE.
Eta was pronounced H in some dialects of Greek, and this is why it has this value in the Roman (and English) alphabet. In other dialects the eta was used for a long E vowel. This is what the Attic (Athenian) dialect standardized on in a spelling reform of about 300 BCE: so it had no way of representing the H sound at the beginning of some words until Aristophanes (not the playwright) remedied it.
He might have been motivated by the fact that the H sound was disappearing, and he was trying to preserve knowledge of the traditional pronunciation. The sound does not occur in Modern Greek (though the letter chi has changed to the raspy H sound of loch, chutzpah), so the choice of breathings makes no difference now. They were official abolished from modern Greek in 1982.
The sound usually derives from an earlier S, which is retained in languages related to Greek: so helio-, hemi-, herpet-, hyper- correspond to Latin sol, semi-, serpent-, super-.
The vowel upsilon (Y or U) always has the rough breathing, as in hypo-, except on the name of the letter itself. All other vowels can occur with either.
The consonant rho was always voiceless at the beginning of a word, as in rêtôr 'orator', and also when doubled the later part was voiceless, as in arrên 'male'. So initially it was given the rough breathing, and in rr it was given smooth on the first and rough on the second. When the Romans transcribed these into their alphabet they used the Roman letter H to reproduce the effect: rhetor, arrhen. This has also disappeared in Modern Greek, where the R is always voiced.
The two breathings are also known by their Latin names spiritus asper (rough breathing) and spiritus lenis (smooth breathing).
The Latin language did not have sounds corresponding to the Greek chi, theta, or phi, so in borrowings they represented these by digraphs ch, th, ph; however in Greek they were single letters different from c, p, t, and had no need of breathings to distinguish them.