Was I wrong in this situation or was the angry bus driver wrong?

Im not great with buses, used to have a phobia due to a traumatic childhood experience, but had to start using them. Today I was waiting at a bus stop and another bus pulled up. It took up all the space so when my bus arrived it was on the other side of the road behind it.

I assumed the first bus would drive away and then mine would pull into the stop near the pavement, but my bus just stopped on the far side of the road for 30 seconds, then drove off.

I ran after it and the driver stopped and let me jump on. He started shouting at me saying "Didnt you see me stop? Why didnt you come over? You should have been paying attention! There was a bus in the way so I couldnt get to the stop!"

He was really angry which upset me but I didnt know you could get on a bus if it wasnt at the bus stop, I thought he was going to come to it once the other bus had moved, and he didnt signal to me or anything.