This command is used in email protocols to begin using Transport Layer Security (TLS) encryption in an unencrypted session. It was originally specified by RFC 2595, "using TLS with IMAP, POP3, and ACAP". An RFC for upgrading SMTP sessions to TLS, "SMTP Service Extension for Secure SMTP over Transport Layer Security" (RFC 3207) has also been published.
However, there can be a security risk to this protocol if it isn't handled carefully on the client's end. IMAP, POP3, etc. were not originally designed to support encryption. To tack on encryption, RFC 2595 specified that servers supporting the protocol would indicate their support by including the word "STARTTLS" in some sort of login banner or capabilities list. A client which is configured to use STARTTLS will, upon seeing that that the server has this capability, initiate the use of TLS by sending the STARTTLS command.
The problem is that an attacker could hijack the session before the client has seen the server's capabilities, or before the server receives the STARTTLS. This would allow the attacker to prevent the client and the server from knowing that the other end of the connection has the ability and the desire to use TLS. What happens next depends on how the client is configured. Here's what the STARTTLS specification has to say on the subject: "Both clients and servers SHOULD have a privacy operational mode which refuses authentication unless [TLS has been activated.] ... Clients MAY have an operational mode which uses encryption only when it is advertised by the server, but authentication continues regardless."
It is that second clause that is the problem with STARTTLS. If clients are configured to do that — to use encryption when it is advertised, but to continue without warning and prompting the user if encryption is not advertised — the attackers will succeed.
If your client doesn't have the capability to warn you when it can't initiate encryption via STARTTLS, you might have an alternative. Some server listen on an alternate port on which all traffic will be encrypted, similar to how HTTP operates on port 80 and HTTPS operates on port 443. When this method is being used, both the client and server are guaranteeing to each other that they understand TLS encryption and are using it.