As an object approaches the speed of light, forces exerted on it will have less of an effect on its motion. Specifically, if one were to try to apply Newton's Second Law of Motion to the situation, it would seem that the object gaining more and more mass as its velocity increased. This is the idea of relativistic mass. According to Einstein's Theory of Special Relativity, the relativistic mass, mrel follows the equation

mrel(v) = γ(v) m0

where m0 is the called the rest mass, which is the ordinary Newtonian mass that you get from applying Newton's Second Law for v<<c. γ(v) is the normal gamma factor of special relativity

γ(v) = (1 - v2/c2)-1/2.

With these definitions we get the following equations for the relativistic momentum, force, and energy.

p = m0 γ(v) v = m(v) v

F = d/dt (m0 γ(v) v) = d/dt (m(v) v)

E = m0 c2 γ(v) = m(v) c2

Looking at the last two equations brings home the principle of mass-energy equivalence in special relativity. The inertia of a body, its relativistic mass, is just the same as its energy except for the extra c2 term.

It should be noted that this term is generally not used in physics today, and "mass" is always taken to refer to the rest mass of an object unless otherwise stated. The term "relativistic mass" is simply redundant. There's no reason to say "relativistic mass" when you could just as easily say "energy" and mean the same thing (except for the c2 factor). Rest mass is an intrinsic property of a body that all observers agree on, much like in Newtonian mechanics. Another reason for choosing this convention is that in General Relativity rest mass and kinetic energy come into the theory in somewhat different ways, so it doesn't really make as much sense to use the idea of relativistic mass there.


Sources:

French, A.P., Special Relativity, W. W. Norton & Company Inc., New York, 1968.
Goldstein, Poole, and Safko, Classical Mechanics, Addison Wesley, San Francisco, 2002.