Group representations occur naturally in applications to
physics because one is always interested in the
symmetry of physical systems. Within group theory they are very important. For example they are the main tool used in the
classification theorem of finite simple groups. So what is a group representation? Before answering that I need a little bit of notation.
If V is a vector space (over a field k) then GL(V) denotes the group of all invertible linear transformations V>V. Note that, if V is finite dimensional and we choose a basis then each such invertible linear transformation can be represented by an invertible square matrix. So such a choice of basis makes GL(V) isomorphic to the group of nxn invertible matrices over k, so you can think in terms of matrices if you wish.
Definition Let G be a group and k a field. A klinear representation of G is a group homomorphism f:G>GL(V), for a kvector space V. The dimension of the representation is the dimension of V. We often refer to V as the representation.
Note that given such a homomorphism G, acts on the vector space by linear transformations. The idea is that our abstract group G which could be hard to understand has a concrete representation by linear transformations (or matrices), which, at least in principle, are easy to understand; we can apply the full power of linear algebra.
Examples

If G=<a> is a cyclic group of order n then we can get a representation of G on a 1dimensional vector space as follows. Choose e a complex nth root of unity then define f:G>GL(C)=C^{*} by f(x^{i})=e^{i}.

The Dihedral group from its definition as the symmetries of the regular ngon has a twodimensional representation.

Looking at the symmetry groups of the Platonic solids gives some
rather nice threedimensional representations of
A_{4},
S_{4} and A_{5}.
Definition If f:G>GL(V) is a representation
and W is a subspace of V then we say that
W is a subrepresentation of V if for each g
in G each f(g)
maps W into itself. Note that in this case, by restricting maps
we have a homomorphism G>GL(W), and strictly speaking this
is the subrepresentation.
A representation that has no subrepresentations except the zero
subspace and the whole representation is called simple or
irreducible.
If f:G>GL(V) and
h:G>GL(W) are representations then we says that
a function T:V>W is a homomorphism
of representations if T is a linear transformation
and T(f(g)v)=h(g)T(v), for all v in V and
g in G. As usual we get a category of representations
and we can prove the isomorphism theorems. If V is a representation
and K and L are subrepresentations then we say that
V is the direct sum of K and L if it is
the direct sum of them considered as vector spaces. That is
if every element of v can be represented as a sum (in a unique way)
of vectors from K and L
For now on we will consider only finite groups and finite dimensional
representations over the field of complex numbers
C. That's not to suggest that there is no interest in representations
of infinite groups
(representations of Lie groups are important in physics)
this restriction is just made for simplicity.
Theorem Every representation is a direct sum of simple
representations. Two representations that are written
as such direct sums are isomorphic if and only if
the same simple representations occur
in each as summands (up to isomorphism) with the same multiplicities.
After the theorem it suffices to classify simple representations
to understand all representations. There is a beautiful theory
contributed to by mathematicians such as Frobenius, Maschke and
Schur that classifies the simple representations using the
character table. One important result is that the number of
simple representations (up to isomorphism) is the number of
conjugacy classes of the group.