Actually, the difference between math and physics is the same as the difference between the English language and a shelf of books. The difference is that you can actually pick up physics and touch it.
Physics is the science of the physical universe, a collection of theories, laws, theorems and axioms that explain how things move, how energy actually works, why the universe is expanding, what everything is made of, and so forth. Today, it's as much theoretical as empirical. But physics, like every other science to a certain degree, is described using the language of mathematics.
Math, you see, isn't about objective truth. Most people think math is somehow intristic to the world around us, when it's actually an invention, something man created in order to quantify the world around us. Mathematics does not deal with reality; it deals entirely with unreal quantities and ideas, with abstract numbers and shapes and forces and formulas.
Mathematics is not tied to any actual things, but it's the language of logic on which all the sciences are built -- and physics most of all.