What It Was
Tanistry was, as Webby tells us, a system of succession that enjoyed long favor and practice in both Ireland and Scotland. The word comes from the Gaelic tana, meaning lordship, and was the way of kings of Scotland from Kenneth McAlpin, the first king of Alba, who unified the Picts and Scots in the 9th Century.
This is as opposed to primogeniture, with which you're probably more familiar and would seem to make more sense--except of course when it didn't.
Why They Did It
The practice preceded the unification of those kingdoms, extending well back into the clan system. Clan leaders were chosen not simply by virtue of heredity, though that did happen, but from a group of viable candidates that were deemed the most capable and best equipped to lead, protect, and improve the fortunes of the clan. The Tanist, or presumptive heir, named by the clan chieftain and/or elected by the full assembly of clan leadership, was usually required to be of full age and possessed of all his faculties. In times of constant turmoil, often too unstable to hand the sceptre of rule down to a direct blood descendant potentially in his minority, the strongest adult male was the best bet.
One can see the benefit of such a system. Under it, the Scottish people were not necessarily subject to the whims of an insane, inept, or tyrannical king simply because he was born to the previous sovereign, and were not at the mercy of bad timing should a monarch die or be killed before his son could safely assume the throne.
However, it also meant that the crown was essentially up for grabs with every generation. Though the king's eldest son was frequently named the Tanist, the practice was not at all proscribed by law and could easily be contested by claims of equal or even greater validity. In addition, the Tanist held the position for as long as the reigning monarch lived; an amibitious heir might seek to speed up the process. Thus the system generally devolved into a Roman Empire-like series of accessions by assassination, full of blood feuds, competition, and an extraordinarily complicated royal family tree. Brother could easily turn against brother, cousin, nephew, mother, sister and so on in order to secure the throne.
Why You'd Know It
In many ways, the Law of Tanistry is at the core of Shakespeare's MacBeth. Though never mentioned in the play, the practice would have been well understood, if perhaps not by the contemporary groundlings, then certainly by the most important person in the audience: James I, who abolished the practice by legal decree early in his reign.
MacBeth was the cousin of King Duncan; both were grandsons of Malcolm II. In 1034, Malcolm was the first king to take a crack at casting off the established Tanist system, naming his son King of Cumberland. In order to make it stick, he attempted to kill off more or less everyone who could have made a competing claim--including the husband of Gruoch. She was the granddaughter of Kenneth III, the king who preceded Malcolm, and the second wife of MacBeth (the nice lady appearing in the play).
The marriage increased the strength of MacBeth's claim to the throne as well as the enmity between the houses. Moreover, according to the Holinshed Chronicle, Duncan was a militarily ineffective king who relied heavily on MacBeth to quash a current rebellion and defeat an invading Norse army--making MacBeth a very likely candidate for Tanist. Duncan's son (another Malcolm) had no automatic right to the throne, but was named Tanist desptie MacBeth's having ostensibly earned the honor.
When James VI of Scotland became James I of England, questions of succession were also pressing on the minds of the populace. Elizabeth I died without issue; James was related closely enough by blood and political sensibility to make a logical choice, and his interest in securing and stabilizing rule made themes of legitimacy an obvious choice for Shakespeare's play. It was important to portray the absolute "good" of succession by hereditary line (Duncan's son becomes Malcolm III) in order to emphasize stability and a clear line of succession--hopefully without bloodshed.
In this respect, the play is a compelling testament in favor of primogeniture and against a "barbarous" Tanist system defined by turmoil and regicide.
But tell that to Charles I.