When he touches you, you could either let him, or tell him to stop, but you know him. If he feels rejected, he'll be upset, he'll throw a fit and
waste so much time and emotion it seems easier to just let him touch you, even though you don't want it. With some people it's just easier to clean up the messes they make than to teach them how to stop making them, or so it seems.
It's like a disease. Assholes are like a disease. It just seems easier to let it happen when it's happening. Normally, you think you have the power to fight it, but when it's there, you just want it to happen and be over with, and if it kills you, maybe that would just be easier.
Sometimes, the cure is communication. Sometimes you can fix the problem with some understanding. Sometimes you don't have to be overtaken and irritated. If you can fix things that way, you're lucky. Most assholes just came into your life and fuck up your day, your week, your month, your head.
Never stop fighting. Never give up on a sickness you can beat and never underestimate what you can beat. Know yourself well and don't give up.