What kind of life do I lead when the reason I can't sleep at night is because I'm not that good at math?

I'm trying to decide if I should drop a math class. It's an elective that I don't need to graduate. Doing the work for the class takes more time than I think it is worth, and I'm not doing that well. Though the class is interesting, learning the material (if I can) will not change my life in any major way. The most I can get out of the course is a slightly more analytical mind, and a few tricks for solving physics problems.

Almost every time I doubt my ability or desire to follow my dream of being a physicist it is because I am having trouble with math. At first I thought it was because I had not learned enough math to be able to do my physics homework. I am starting to realize, however, that the problem is more fundamental than that. I know a lot of math. It is the purely analytical nature of the subject that I have trouble with. I usually do not care about proofs or derivations of theorems or formulae. This is a problem.

If I do not care about how the very basis of physics works, then I do not care enough about physics. Not enough to go to graduate school. Not enough to spend my life thinking about it.

Thoughts like these used to make me physically ill. Now they only keep me up at night.

I'm dropping the class.

(Sorry if this was boring, but I really need to get to sleep.)