The origin of most of the numerals is obvious: I is a
finger, V is a
hand, C stands for
centum, and M stands for
mille. I don't know where L comes from, but D for 500 is apparently derived from half a thousand. It uses a sign we no longer have in 'modern' Roman numerals, the
apostrophus, but which had several uses in the early system. This was a reversed C. If I represent it imperfectly by a closing bracket, then I) was 500 and CI) was 1000. The (symmetric) sign CI) could well have arisen from a rounded letter M for
mille 'thousand', then the pieces disassembled to give the half-a-thousand sign I), later written as D. I believe this is correct but cannot swear to it that I) was prior to D.
The I) symbol was multiplied by ten by adding apostrophi: I)) = 5000, I))) = 50 000. Adding as many C's in front doubled them: CI) = 1000, CCI)) = 10 000, CCCI))) = 100 000, and so on. (These would definitely look better if you envisaged them with the reversed C.)
One ancient usage that has survived is that of IIII instead of IV on clock faces.
In the Middle Ages they sometimes used the rest of the alphabet for extra numerals. This appears to have been a pretty slapdash requisition, as there are repeats and gaps, and the order doesn't make much sense. I have no idea how widespread this usage was, if at all, or whether it was completely esoteric.
S = 7 or 70, O = 11, F = 40, A = 50 or 500, S = 70 or 7, R = 80, N = 90, Y = 150, T = 160, H = 200, E = 250 and K = 250, B = 300, G = 400 and P = 400, A = 500 or 50 and Q = 500, Z = 2000.
Over-barred letters for thousands are a mediaeval invention: I'll represent them here as e.g. V for 5000. If my source (below) is to be trusted not to contain misprints, several anomalies included B for 3000, A for 5000, and S for 70 000 (not 7000), and the lack of K, Z.
Millions were indicated by putting vertical bars around numbers, starting with either |X| or |X| for 1 000 000.
They also sometimes reversed D, with the same value. The apostrophus could be written small, which I'll show that here as o: an example is from a printed book of as late as 1704, where the date is written cIo Io CCIV.
With the development of lower case letters these were also used in numerals. When two or more i's occurred together the last of them was often written as j, as in xiij '13'. I and J were treated as the same letter until about the eighteenth century.
All above gleaned from Chambers Dictionary
Another fun thing they used to do with Roman numerals was numerology, in the obvious way: add up the classical ones and ignore the rest, so just find a name containing DCLXVI and you're laughing. Or your head is swivelling round in a circle.