Did I do the right thing?

So just right now I was at Wal-Mart looking at a sale on pasta. All of a sudden, my cart gets pushed by some home-boy punk. I just up and realize what happened, then just put my head down and think, "what an idiot." Then he comes right up to me and says, "whats up dog you were in my way." I wasn t expecting this at all. I then said, "sorry man, but you didn t have to be an *** and push me out, you could have said excuse me and I would have moved." He then like an idiot kicked my cart and said, "what are you going to do about it *****." I smelled so much weed on this guy, I knew he was slow and wouldn t last in a fight. As for me, I m an amatuer boxer with an unofficial record of 11-0 and 4 KO s at 145 lbs and at 5 6, shorter guy. He was about 5 11, 200 lbs with some muscle, but high and no way I would have loss. I then said, "you know what man, your not worth it," and he walked away. I ask if I did the right thing because he s a bully and I could have hurt him no problem out in the parking lot at 9 o clock at night with little people to see it. I work for the courts and know the police, judges, and attorneys around the area. They would have sided with me, but I didn t fight him. Should I have taught him a lesson, or was I right in just letting it go?