The question is: Is it is more damaging to use a symbol we attribute power to (such as a flag or swastika) in a "negative" way than it is to deny citizens that right?
I suggest that once laws are in action that prohibit activities that cause no DIRECT harm to another human being, the line between personal freedom and "the good of the whole" becomes dangerously fuzzy.