Out-of-band data is used with network streams (like TCP) to send a message with a higher priority than the rest of the data. It can be used to signify the other side of an error, or any other important event, which has to be processed immediatly.
A popular, though not quite friendly, use for OOB data is to send something to port 139 (netbios) on a Windows computer. This attack will effectively "crash" the system. More commonly known as "winnuke".