In the publishing industry, the terms 'boutique imprint', 'boutique press', and 'boutique publisher' are used to indicate either a small press or a small imprint of a larger publishing company. In the case of boutique imprints, these can be extremely popular and successful subsidiaries of larger companies - or they can be no-name subsidiaries of no-name publishers.
The label 'boutique' doesn't really mean anything specific, and boutique imprints cover a wide range of publishers. Dutton Books is a boutique imprint of Penguin Books, which in turn has the subsidiary Dutton Children's Books. It turns out approximately 120 books a year, many of them very successful. Angry Robot Books is a boutique imprint of Harper Collins, which turns out 25-30 new titles a year, of moderate-to-low popularity. Many small publishers have small subsidiaries that cater to very specific interests, and these subsidiaries are nearly always called boutique imprints.
This might lead one to believe that the term 'boutique imprint' is redundant, but while it is vague, it does have some meaning. There are many imprints that are too big to be considered 'boutique'; for example, Random House has the imprint Ballantine Books, which in turn has the imprint Del Rey Books, which in turn has the imprint Del Rey Manga... none of which are small enough to be referred to as boutique imprints.