Mathematics about mathematics. A branch of
maths and
philosophy dealing with the
foundations of mathematics. Includes
formal systems,
logic,
mereology, much of
ontology, parts of
metaphysics, etc. Nowadays, many people would include
model theory,
category theory and
topos theory (the latter being a branch of category theory relating mathematical systems to certain kinds of categories called
topoi).
Russell and Whitehead's Principia Mathematica is metamathematics. Gödel's famous On Formally Undecidable Propositions of Principia Mathematica and Related Systems is metamathematics. Frege, Wittgenstein, and many others are famous for their metamathematics (among other things, of course).