multidimensional normal distribution

We can use an amazing fact about the normal distribution to define multidimensional normal distributions in an elegant way. To pick a d-dimensional normal distribution, just pick each orthogonal coordinate independently according to a normal distribution (the coordinates may have different standard deviations, if you like; the claims made in the sequel still hold).

Obviously, this defines a distribution on R^d. What is less obvious is that this isn't dependent on the specific set of coordinates chosen! But it turns out that if you pick some other orthogonal basis, and take coordinates according to that, the projections of this distribution on each new coordinate are normally distributed, and independent!

This situation is truly impressive (for instance, if we used uniform distributions instead of normal ones, we'd still get a uniform distribution on some box in the first step; but the axes of the box would be parallel to the axes used, and any rotated set of axes would not have uniform or independent projections!

The probability density function for the d-dimensional normal distribution with mean vector μ and covariance matrix Σ is given by the formula:

p_μ,Σ(x) = (e ^{-(x-μ)TΣ-1(x-μ)/2} ) / ((2π)^d|Σ|)^1/2,

where x and μ are d-dimensional column vectors and Σ is a d×d matrix.

If the covariance matrix is singular, the naive application of this formula will run into problems, as in that case Σ will not be invertible and have a determinant of 0.

The singularity of Σ implies that the normal distribution is flat along one or more directions. The distribution is not really d-dimensional - it's less than d-dimensional. Specifically, the distribution lives in an affine subspace of the d-dimensional space. The rank of Σ gives the dimensionality of that affine subspace.

So the trick to handling singular Σs is this: first check if x is on that affine subspace. If it's not, then the probability density is automatically 0. If it is, then we reduce the dimensionality of the problem and evaluate the density on the affine subspace (Hint: use the SVD of the covariance matrix).