Mathematics about mathematics. A branch of maths and philosophy dealing with the foundations of mathematics. Includes formal systems, logic, mereology, much of ontology, parts of metaphysics, etc. Nowadays, many people would include model theory, category theory and topos theory (the latter being a branch of category theory relating mathematical systems to certain kinds of categories called topoi).

Russell and Whitehead's Principia Mathematica is metamathematics. Gödel's famous On Formally Undecidable Propositions of Principia Mathematica and Related Systems is metamathematics. Frege, Wittgenstein, and many others are famous for their metamathematics (among other things, of course).