To Gritchka's excellent historical survey I might add a comment or two from the perspective of an aspiring pure mathematician.

A few decades ago it was fashionable to say that "set theory is the machine language of mathematics" -- that is, that ultimately all mathematical statements have force by virtue of their theoretical possibility of transcription, "compilation" if you will, into propositions of set theory. The benefits of this reductionist approach are obvious: axiom systems for set theory are short, and if you accept an axiom system, you are forced to accept any statement which can be compiled into it -- never mind the improbability of actually doing this compilation, as mathematicians are already accustomed to nonconstructive existence theorems. (Some people have done interesting combinatorial research on the rate of growth of proofs as they are reduced to axiom systems.)

However, there are several axiom systems you might choose: set theorists have standardized on ZFC for the most part, but category theorists routinely require the notion of a proper class (a class which is not a set) and therefore must work in a variation of Gödel-Bernays set theory. Working mathematicians not interested in the foundations of mathematics usually ignore all these issues and use something much less formal which probably resembles the Kelley-Morse variant of GB.

All these systems are formulated in classical logic, but what if your work requires another logic entirely? The semantics of type theory, for instance, closely resembles certain intuitionistic logics, and those interested in quantum mechanics or quantum computing need yet another group of logics. A set theory formulated in a nonclassical logic would be entirely different from the standard ones.

To me, however, the most convincing objection to the set-theory-as-primary approach is that mathematicians don't work that way. We have things in mind that we want to use mathematics to deal with, and we treat the set theory, the category theory, the logic, the physical world, or something else as primary depending on what works best. And when we want to study a particular theory, we need to build a metatheory around it anyway; so the better way to proceed is to be able to describe all of our theories in terms of each other.

These thoughts come largely from Paul Taylor's excellent book Practical foundations of mathematics (in the series Cambridge studies in advanced mathematics), which I encourage anyone interested in its topic to read, after getting some familiarity with basic classical logic. For that you might turn to Mathematical logic by Ebbinghaus, Flum, and Thomas.