What should I have done?

I was at a concert a couple of weeks ago.  Obviously, a crowded place, with lots of people very close together.  One particular man was making me uncomfortable.  He then escalated his behaviour, actually rubbing against me, and starting to breathe very heavily in my ear, along with some very peculiar groaning noises.  Obviously, very unpleasant, but I didn’t want to leave the concert to find security, to then get into an argument about whether his behaviour was inappropriate or I was just over-reacting.  So I made my boyfriend change places with me and got on with enjoying myself. 

I still feel a bit guilty about not reporting this behaviour or making more of a fuss.    I felt that by letting it go unchallenged, I was excusing him.  I also worry that by letting this go, he’d continue to escalate this behaviour thinking it was ok.

But I’m still, weeks later, really not sure how else I could have dealt with this except moving away.  It was a loud, crowded venue, so telling him to F*&k off would have probably engaged me in some “I didn’t quite hear that” conversation with someone I really didn’t want to speak to.  Leaving to find security would have meant I’d miss the concert, as well as then starting a “he said / she said” argument that probably wouldn’t have acheived anything.

But at the same time, I’m incredibly angry that this man got to treat me like that without consequence.

I would really value any opinions on what I should have done – I hate the idea of living in a society where the best course of action is just to ignore it and move away, but I still haven’t thought of a satisfactory and constructive way that I’d like to have dealt with this differently. What do you think?