The
covariance matrix of an
n-
dimensional vector random variable X is the
n×
n matrix of the covariances between the elements of the
vector. The covariance matrix is the
generalization of the
variance of a
scalar random variable to multiple
dimensions.
The covariance matrix is the measure of how spread out the probability distribution of X is in n-dimensional space. The 'larger' the elements of the covariance matrix, the more spread out X is.
The covariance matrix of X, often denoted Σ, is defined by the formula
Σ = E[(X-μ)(X-μ)T],
where
X and
μ are
column vectors and
μ = E[
X] is the
mean of
X.
The element of the matrix at row i and column j is the covariance between the ith and jth elements of X. Specifically, the ithdiagonal element of Σ is the variance of the ith element of X.
The covariance matrix is always symmetric and positive semi-definite. These facts can be proven easily from the definition.
The covariance matrix may be singular. This can happen if the variable X does not have any variation along one or more dimensions.
More generally, the singularity of the covariance matrix implies that the distribution of X is flat along one or more orthogonal directions in n-dimensional space. The distribution is not really n-dimensional - it's less than n-dimensional. Specifically, the distribution lives in an affine subspace of the n-dimensional space. The rank of Σ gives the dimensionality of that affine subspace.