Would you tell if this happened to you?

I don't know what to do. I was at my sisters house last weekend and I slept on the couch and at 3 in the morning i woke up to see her boyfriend in the livingroom watching tv and I started watching with him but he was drunk and we were joking around and i was falling asleep off and on so the next thing i saw he sitting next to me on the floor while i was lying on the couch and i felt the covers moving and his hand moving up my thigh it freaked the heck out of me cause ive known him forever like a big brother and he has never done anything like that before so when my sister came in i didnt say anything cause i know how much she loves him and i didn't want her to blame me for him wanting to do that to me so was i wrong to leave it alone? plus i really like him as a person and dont want to be the reason they break up