I very recently found out, at the bright young age of 26, that my inability to enter a mall or complete lack of desire to go to a public place with any form of lighting is called "agoraphobia". I've only spent the past twenty or so years avoiding all people because I didn't want to be my plebian needs and emotions to be a bother to them.
Low self-esteem isn't selfish. It's almost selfless how I torture myself into not speaking because I feel my voice is insignificant and everyone, I'm sure, has more important things to say. No, I've learned to stop wanting friends. At the moment, I'll settle for someone knocking on the door of my hobbit hole in an attempt to sell me a religion.