Generally refers to the relatives
outside of one's own immediate mom-dad-brother-sister family, although rather impolitely. The word rellies (always plural, never singular) implies the massed gibbering hordes
of aunts-uncles-cousins-nephews-nieces whose strange quirks
and embarassing personal problems
(as opposed to the normal quirks
and mundane personal problems
of your immediate family, which are perfectly ordinary) cause you to tell your friends that they can't come over this Thanksgiving
weekend because you're sick.
Interestingly enough, rellies usually only show up during the holidays, and always in large packs. If your uncle arrives for the weekend, he's just your uncle, but if he brings his wife and their three daughters with, they become rellies and all hell breaks loose.
Each of your rellies has a strange behavior that stands out even among all of their others. This behavior will destroy your sanity if you stay near the perpetrator too long. Examples would include Uncle Bob's incessant chewing tobacco habit, Cousin Alice's love of shouting the lyrics to her Britney Spears CD, or Auntie Gritchka's insistance that your videogame music is driving her mad and would you please shut it off and those horrible games rot your brain besides. The primary quirk is the one that will most irritate you and will be the one you remember for months to come when said relative is mentioned. Strangely, these behaviors never seem quite as bad in retrospect, a misjudgement that will certainly haunt you the next time they come over.
Of course, no one would ever consider your immediate family rellies. All of your quirks and wacky habits are lovable and entertaining.