Mathematicians often chide Physicists by saying that physicists are limited by nature whereas mathematics breaks these shackles and is the only pure truly pure subject. Is this really true?

I know this sounds weird but consider the natural number system. When you say that the set N has no upper bound the basis behind this statement is that given any number you can always conceive of a number larger than that.
For example given a number n=10^(10^(10^10)) you could think of the number n+1. However is this ability not limited by the human power of conception. And is this not an aspect of nature?

For example if we were to find a species of aliens who could simply not conceive a number greater than 10 because they had some peculiar mental block, mathematics would take on an entirely new meaning. So how far do we really escape from nature with mathematics