A system is said to be at an unstable equilibrium when a slight pertubation will knock it out of equilibrium. For example, a ball balanced on the top of a bowl is at an unstable equilibrium because any slight force that acts on the ball will cause it to roll off. Compare with stable equilibrium, an example of which is a ball at the bottom of a bowl. Any force on this ball and it will roll around a little bit, but then come back to rest at its initial position.
In Classical Mechanics, an unstable equilibrium is at a point where
-- = 0
--- < 0
where U is the potential energy.