As an object approaches the speed of light, forces exerted on it will have less of an effect on its motion. Specifically, if one were to try to apply Newton's Second Law of Motion to the situation, it would seem that the object gaining more and more mass as its velocity increased. This is the idea of relativistic mass. According to Einstein's Theory of Special Relativity, the relativistic mass,
mrel follows the equation
mrel(v) = γ(v) m0
where m0 is the called the rest mass, which is the ordinary Newtonian mass
that you get from applying Newton's Second Law for v<<c. γ(v) is the
normal gamma factor of special relativity
γ(v) = (1 - v2/c2)-1/2.
With these definitions we get the following equations for the relativistic momentum, force, and
energy.
p = m0 γ(v) v = m(v) v
F = d/dt (m0 γ(v) v) = d/dt (m(v) v)
E = m0 c2 γ(v) = m(v) c2
Looking at the last two equations brings home the principle of mass-energy equivalence in special
relativity. The inertia of a body, its relativistic mass, is just the same as its energy except for
the extra c2 term.
It should be noted that this term is generally not used in physics today, and "mass" is always taken
to refer to the rest mass of an object unless otherwise stated. The term "relativistic mass" is simply
redundant. There's no reason to say "relativistic mass" when you could just as easily say "energy" and
mean the same thing (except for the c2 factor). Rest mass is an intrinsic property of a body that all observers agree on, much like in Newtonian mechanics. Another reason for choosing this
convention is that in General Relativity rest mass and kinetic energy come into the theory in
somewhat different ways, so it doesn't really make as much sense to use the idea of relativistic mass there.
Sources:
French, A.P., Special Relativity, W. W. Norton & Company Inc., New York, 1968.
Goldstein, Poole, and Safko, Classical Mechanics, Addison Wesley, San Francisco, 2002.