Turing-complete languages or machines are those that can express arbitrary computations, that is, they can emulate arbitrary Turing machines.

This is not at all the same as the capability to solve any mathematical problem: many perfectly defined mathematical problems are provably unsolvable.

As a matter of fact, the Turing machine was invented to prove this.

A simple Turing-complete formalism is, obviously, the Turing machine; another, the lambda calculus; another is the arbitrary rewrite grammar often used as the last formalism in the Chomsky hierarchy.

Very small subsets of most programming languages are already Turing-complete, on the condition that they can access arbitrary amounts of memory. C, for instance, isn't, because it must address all memory with pointers of a fixed size; but a variant that uses pointers of arbitrary size would be trivial to define.