Reluctantly, I'm in favour of hate crime legislation. It's been shown that hate crimes have a more severe and lasting psychological effect on their victim than if they'd suffered the same crime for another reason. You're more likely to suffer Post-Traumatic Stress Disorder, for example.
We punish other crimes more severly if they affect the victim more severely. Attempted murder carries a lower sentence than murder. Like Jackie Mason said: 'They're being penalised for what? Competence?' No. They're being penalised out of society's need for revenge, because the person is gone, and can't enjoy their life any more, and society doesn't have them any more.
So we punish people who commit hate crimes more severely, because they've damaged their victim more, and society more. It's not because of what you were thinking, bigot, it's what you *did*.