"Noisette" is also the French for hazelnut; presumably, the French gardener and hybridist mentioned by Webster 1913 had this in mind.

A "noisette" is not a small noise of female gender, however tempting it may be to assume so.