I've been wondering about something like this for a long time, and I have a theory.
If a person perceives colors differently from the rest of the world, they'd never know it. I mean, that's how colors have appeared to them their whole life. If they see a color that they know to be red, and someone else looks at the same red and sees something different, neither person would be aware they're seeing anything different than the other person. Each would assume the other sees the same thing.
What I'm saying is, both people can look at something and each person sees a totally different color than the other. One person says "That's red." The other person says "I know."
Therefore, it's possible that your perception of the color red is totally different than another person's perception. However, it's universally agreed to be red, because that is the association you have made with that color for your entire life. Back in your early years, someone pointed to that color and said "Red", and you made the association.
This means that it's possible that everyone sees colors differently. What looks red to you might look green to someone else, and you both would never know it.
Noder's Note: I'm not saying this is the way it is. It's merely a possibility.