I've been building my own personal library since I was about 7. I think.
Admittedly, the majority of novels I own are of the speculative fiction genre, but I also own a shelf of plays, a number of books that I couldn't sell back from college, and a rather obnoxious number of comic books.
Personally, I've been inspired by tales of Robin Mckinley's personal library, which apparently numbers over 10,000 books. I'm only at around 900, myself. Not counting the comics. *grin*
I think that the impetus, quite often, to build a personal library is not just bibliophilia, but also a need to have the entire series of books (rather than book 2, 4, 5, and 7 like most libraries seems to do) and the ability to read a book yet again on a whim.
I certainly couldn't imagine life without it, despite the onerous pain in the ass it is to move them.