Putting a penis in your mouth and stimulating your partner to orgasm is a hell of a lot more intimate than a handjob or mutual masturbation or a peck on the cheek. To Clintonically claim that it isn't sex demeans both people in the exchange.
That said, there's nothing wrong with casual safe sex, as long as all the participants are consenting adults and aware that the zipless fuck is the order of the day.