Before Martha's Vineyard was
posh it was
deaf. Martha's Vineyard used to have one of the highest concentrations of deaf people in the world. Some of the first European families to settle the island brought with them a
recessive gene causing
hereditary deafness. Subsequent generations of
inbreeding (it was an island, after all) produced a population that was about 25% deaf.
But unlike most deaf people of their time, these people were educated. This was pretty unusual in the nineteenth and early twentieth centuries, a time when to be deaf was often to be painfully ignorant and often entirely without the use of language. Almost all the deaf residents of Martha's Vineyard were sent to the nearby Hartford Asylum For The Deaf to be educated. And lucky for them, they went to a school that believed in sign language, not just lip reading.
Not only were the deaf people of Martha's Vineyard educated, they had a larger community with which to interact. Because almost everyone on the island had a deaf family member, almost everyone on the island knew how to use sign language. Which meant that all the residents of the island, deaf and hearing, could communicate with each other freely.
So what do you get when you have generations of people (and educated people at that, who had large vocabularies and could read foreign languages, liked to engage in political debate and discuss current events) all communicating with each other in what had started as a somewhat limited signing method imported from France? You get the emergence of a robust, full-fledged language, full of metaphor and idiom, perfectly capable of making "infinite use of finite means." (Humboldt's definition of what counts as real language.)
And when these people go back to the Hartford Asylum, where families from all over North America started sending their deaf children to be educated, and those children are bringing sign back with them to their own deaf communities, you get the emergence of a national, unified, American Sign Language.
Sources:
Groce, Nora Ellen. Everyone Here Spoke Sign Language
Sacks, Oliver. Seeing Voices