The following is excerpted from the notes for a discrete mathematics course that I am writing. My main reference while writing most of this material was Roads to Geometry by Wallace and West.

We create an axiomatic system in order to provide a framework for our mathematical notions. By carefully choosing axioms and restricting ourselves to working within this system, we ensure the truth of all subsequent conclusions.

An axiomatic system consists of

Undefined terms
In and of themselves, these terms are meaningless placeholders. From a practical standpoint, they are chosen to help us analogize the idea being represented in our mathematical system. For instance, we take the notion of a point in geometry to be a dimensionless entity, the smallest available unit of geometrical space. From an axiomatic standpoint, however, the term "point" is meaningless and remains undefined.
Axioms are statements describing how the undefined terms interact. We take these statements to be true without proof. These are the basic "rules of operation" of our system.
A definition assigns a convenient placeholder name to a complex idea. The meaning of the idea is not changed through the assignment of a definition.
Theorems are true statements in our system that are logical consequences of the above entities.
It should be noted that every true statement in our system should fall into one of the latter three categories above.

The desired characteristics of our axiomatic system are as follows. They are given roughly in order of importance, with the first being essential to a well-defined and useful mathematical system and the last being almost totally optional.

The axioms we choose for our system should not contradict each other. If they did, we would be able to prove as true a false statement. This is undesirable, for, as we shall see, the existence of a contradictory conclusion in our axiomatic system implies that all statements can be proved true, for example, "All odd numbers are prime" or "If 16 is a perfect square, then Paris is the capital of Mars" or "All work and no play makes colorless green ideas sleep furiously." For more information on this, see vacuous proof.
We should have in our set of axioms all statements that are not somehow equivalent to the existing axioms and that don't require additional undefined terms. This sounds pretty complex, but what it is basically saying is that we shouldn't be able to come up with a statement in our system whose truth or falsity we are unable to determine. This is a reasonable request--that every statement we make in our system is either true or false.
The axioms we choose should introduce sufficient structure that we can talk about any mathematical idea that we will come across (depending on the context in which we are working). For most mathematical purposes, this means that the system needs to be rich enough to solve polynomials in the complex numbers. This is not to say there are no useful systems that are less complex than an algebraic one, but most "interesting" mathematics requires a sufficiently complex system to model it. As it turns out, there is a slight problem with any system complex enough to contain arithmetic.
The axioms we choose should not be redundant. That is, none of the axioms in our set should be a logical consequence of the others in any combination. By keeping our axiom system to a minimum we ensure that we have made as few assumptions as possible. There is nothing inherently wrong with a dependent set of axioms; in fact sometimes it's advantageous because it can help us avoid a lot of tedious foundational work. But the ability to state an equivalent system in an independent way is desirable.

A convenient way of expressing the first two rules is as follows:

  • We shouldn't be able to prove false statements as true, and
  • We should be able to determine whether any given statement (of unknown truth value) is false or true.

The crux of the problem with any axiomatic system lies in the combination of the first three of these properties. Godel's Theorem states that any "sufficiently complex" system is either incomplete or inconsistent, and worse, there is no way to know a priori which is the case. As mentioned above, "sufficiently complex" generally means "strong enough to contain arithmetic". So for such a system, either the system harbors a contradiction, or there are valid statements whose truth value cannot be decided.

For simpler systems, Godel's theorem does not apply. There are several useful systems that are not complex enough for Godel's Theorem to affect them: for example, the propositional calculus is quite useful, but it does not have enough structure for arithmetic to be performed (as it turns out, within its context it is both complete and consistent). Once you add in set theory, however, you're in trouble. To go into too many details about Godel's Theorem is outside the scope of this writeup; there are several good writeups there if you would like to explore this idea further.