The study of derivatives (including antiderivatives) to fractional orders. For instance, rather than the second derivative of sin(x), you would take the halfth derivative of sin(x).
Denote the nth derivative Dn and the n-fold integral D-n. Then
D-1f(t) = ∫0tf(ξ)dξ
D-nf(t) = (n-1)!-1 ∫0t(t-ξ)n-1f(ξ) dξ
is true for n, then
D-(n+1)f(t) = n!-1∫0t(t-ξ)n
But the former equation is true for n = 1, so it is also true for all integral n by induction.
The fractional derivative can only be given in terms of elementary functions for a small number of functions. For example:
D-νt-λ = Γ(λ+1) / Γ(λ+ν+1) tλ+ν
Where Γ(x) is the Gamma function. (CRC Concise Encyclopedia of Mathematics, Eric W. Weisstein)
In practice, the fractional calculus is not a very useful tool (except perhaps on a theoretical level), and is a pure figment of recreational mathematics.