Handshaking protocol is a series of steps used to coordinate asynchronous bus transfers in which the sender and receiver proceed to the next step only when both parties agree that the current step has been completed.
At a more detail level handshaking is when data is transferred from the hardware port to the receive buffer. When a character of data arrives at the serial port, the communications device has to move it into the receive buffer so that your application can read it. If there is no receive buffer and your application is expected to read every character directly from the hardware, you will probably lose data because the characters can arrive very quickly.
A handshaking protocol ensures that data is not lost due to a buffer overrun, where data arrives at the port too quickly for the communications device to move the data into the receive buffer.