Venery is a fun word, because it's got two unrelated etymologies: When it refers to hunting, it's derived from the Latin "venari", to hunt. When we're talking about sex, it's derived from Venus, the goddess (allegedly) of Love.
At the moment, we're talking about hunting.
Nicholas Cox1 defines "beast of venery" as a "term of art" referring to "the Hart, the Hind, the Hare, the Boar, and the Wolf." A hart is a boy deer (see also Heorot, saith Webster); a hind is a girl deer. A hare is a rabbit, a boar is a wild pig, and I hope you already know what a wolf is. Cox credits this list to "Budaeus... in his Treatise of Philology". Cox differentiates these critters from the Beasts of the Chase on the grounds that the former spend their days in "the great Coverts and secret places in the Woods", sneaking forth to nibble on people's lawns at night, while the latter "do reside all the day-time in the Fields, and upon the Hills or high Mountains".
Who, then, are the beasts of the chase? "The Buck, the Doe, the Fox, the Marten, and the Roe". Now, a buck is a boy deer and a doe is a girl deer (sound familiar?), and a roe is a girl deer yet again -- unless he refers to fish eggs, but those are neither a "beast" nor often found "upon the Hills or high Mountains". He might also mean a roebuck, but that's just one more kind of deer. Isn't that odd? Three of the animals in this list are the same animal as each other, and they are also the same animal as two of the animals on the other list.
I don't believe that Mr. Nicholas Cox knew that a doe is a roe is a hind. I don't think he ever turned his hand to hunting harts and martens, and I have no faith in his sources, either. This stuff is all charmingly archaic, but it's not real information. It's one writer with no real knowledge quoting another -- who got his facts, such as they are, from a third. It's scholarship, as opposed to knowledge.
When we cast a wider net, the waters grow muddier yet: William Twici2, chief huntsman for Edward II (reigned 307-27 AD), is said to have classified the fox as a beast of venery3. The term seems also to have been used in the Forest Law (1072 AD) of William I, often called William the Conqueror. In a nutshell, the Forest Law invented poaching by criminalizing the hunting of certain beasts in certain forests (royal game preserves, if you like) by anyone not authorized.
I'd take William Twici's word on the fox issue, as he seems to have been a pro, but did anybody really bother hunting foxes back then? The king's huntsmen were scrounging up dinner for the quality, and foxes aren't edible. Let's play it safe: A beast of venery is, at the very least, one which is hunted.
Why do we care? Well, in Chapter III of Sir Gawain and the Green Knight, we see a reference to "beasts of venery, and beasts of chase, and all manner of vermin"; they gallop past us again in Le Morte d'Arthur (Book VIII, Chapter III, and Book X, Chapter LII).
1Nicholas Cox,
The Gentlemans Recreation, in four parts: Hunting, hawking, fowling, fishing.
Published by Maurice Atkins, London 1674
http://homepages.ihug.co.nz/~battle/text/animals/huntsman.htm
2That name sounds familiar, somehow.
3This is according to http://65.1911encyclopedia.org/H/HU/HUNTING.htm.