In his book The Pictoral Guide to the Living Primates (1996), Noel Rowe defines noyau to be "A social structure in which a male's territory overlaps the small territories of several females; found in several nocturnal promates."*
As was said above, this behavior is found in many nocturnal primates, including the calabar potto and the orang-utang. Characteristics of this structure also include casual, transient mating between the primates, and the young being raised with the mother. Contrast with tribal.
The plural of noyau is noyaux.
Gordon R. Dickson, in his book Wolfling, defines noyau to key on boundary posturing and other acts of territorial defense. While this can be a property of a noyau culture, it is not required.
Incidentally, noyau is French for "pit of a fruit" and is used to refer to the Linux kernel.
* All of the quoted definitions I've found either cite Rowe, or give an identical (word by word) definition. If you have any insight into the origin of the term, /msg me.