| According to the A.Word.A.Day mailing list, a "brown study" is a state of absorption in thought, from the idea of brown being gloomy. The Oxford English Dictionary online adds "especially an idle or purposeless reverie" and records the phrase as far back as 1532.
The word "study" has an old meaning of anxious thought or abstraction, but its use in this phrase makes me think of the artistic uses of the word, "an object presenting effects of colour (and the like) attractive to an artist" and "a literary work executed as an exercise or as an experiment in some particular style or mode of treatment." (OED) My interpretation is that someone in a brown study could be the subject of a painting done in shades of brown. | 
