Prefaced by, "Let every man mind his own business, and endeavor to be what he was made. Why should we be in such desperate haste to succeed and in such desperate enterprises?", the actual quote from the Conclusion of Thoreaus' Walden is:
"If a man does not keep pace with his companions, perhaps it is because he hears a different drummer. Let him step to the music which he hears, however measured or far away."
The above quote was taped to my refrigerator for years, courtesy of a first grade teacher, plus on a wooden plaque in a bathroom at rising eye levels, as my sons grew. I re-read the slim, musty paperback, from a distance of forty years, another distance from its publication date in 1854.
As with many great quotes, when tinkered with or taken out of the context in which they were written, the meaning can become distorted to suit individual beliefs or groups needing a slogan to rally their cause.
The text before and after add layers of richness one cannot ignore. This phrase, often used to promote individuality seems not to have been the intent of Thoreau, but rather being content with one's life and using your talents in tandem with "the Builder of the universe".
"The fault-finder will find faults even in paradise. Love your life, poor as it is. Every nail driven should be as another rivet in the machine of the universe, you carrying on the work. Rather than love, than money, than fame, give me truth."
My understanding of the original phrase is less about being different yourself and more about allowing others to live to a music you cannot hear.