Ranking, or trading stylized insults, is an African-American tradition. The insults are sometimes in improvised poetry or a singing duel. You can hear it in rap music, but it is older than that; it is also called doing the dozens, playing the dozens, signifying, or snapping.
In a ranking contest you might mock your opponent's mother, their girlfriend, their lack of sex life, their silly appearence, and their stupidity.
Signifying implies more cleverness and trickiness- you might be mocking someone subtly or indirectly in a way they don't understand.
The tradition of ranking contests is why a lot of rap songs consist of boasting and insults.