Yes, I've had similar experiences, also being skeletally thin all my life. I tend to look like I've been starved for a few weeks too long, and when I inhale I can look positively emaciated. I was "stringbean" or "spaghetti" in elementary school, or other names folks could think of. Coupled with my generally non-athletic demeanor and personality, I was an easy target in fourth grade... In High School, when changing for gym class, I got dubbed "The Dachau Kid," in a show of incredibly poor taste (this was at a Jewish High School).
On the other hand, people watching me eat have commented that I look like I haven't eaten in weeks. I can fast for a day without much trouble, or skip meals with impunity, but when the mood strikes mind and/or body I can annihilate rather frightening amounts of food in very short order. I haven't gained much weight in years; maybe ten pounds in the past ten years. I don't gain or lose much, as a rule. I don't exercise much, and even when I did it didn't show. I got stronger, but you couldn't tell it from looking at me, unless you looked closely.
No, being skinny isn't all peaches and light for a guy: you can be too thin. I think I have to admit I'm happier this way than the opposite, though, and there's something to be said for being unusual in a world of normalcy, so I'm not sure I'd be happier being "normal" either.