There's a rather simple explanation for the societal acceptance of innumeracy: math isn't as necessary for everyday life at literacy. Sure, you might lose a bit of cash at the grocery store or at the theater, but all in all, you won't be too much worse for wear. Being illiterate poses another problem entirely. Imagine not being able to read road signs, store signs, menus, or simple instructions. Besides factory work, you wouldn't be able to do much else in our current society. Secretarial, data entry, and even fast food jobs require at least some rudimentary knowledge of reading and writing.

Okay, the above attempted to explain the why it's socially acceptable to be ignorant of math, but it didn't really explain the pride through ignorance part. Don't worry: I've got a pretty damn good hypothesis for that, too. For some reason--at least in American culture--mathematics is seen as a difficult field of study. The only problems are that math is a very, very large group of concepts and that many people find it easier to generalize than to specify. Sure, PhD-level graph theory may be agonizing for just about any living person, but simple arithmetic and algebra should be easy for anyone, regardless of age, to grasp. Herein lies the problem: all three of these are still math. Because people like to generalize, a quadratic equation appears to be just as difficult as really nasty differential equation shit (does it show that I'm not a math person?).

By now, you're probably thinking "Why don't I downvote this bitch right now? He's not making any sense," but this is where I'll tie it all together. Academically challenging studies are often attached to the geek stereotype, and until recently, being a geek was not a good thing. Essentially, by saying "I'm as dumb as an ox when it comes to math," one was saying "I am not a geek. I fit in with society. I am not a misfit." Keep in mind that math is hardly the only field where this applies: physics, computer science, organic chemistry, and rocket science (aerospace engineering) were all subject to the same social stigma. See siren's writup for a perfect example. By distancing himself from intellectual fields, a man believes that he is gravitating towards the norm. For many people, being perceived as normal is extremely important.

As an aside: For some reason, doctors and lawyers escaped this stereotype, despite their studies being considered difficult. I think it has to do more with the fact that both professions seem to have more societal interaction, therefore escaping the anti-social geek profile.