How to deal with this guy at work I REALLY want to just fight?

Most the people at my job are either friendly or do-able. There's this ONE guy, ONE guy at my job who thinks he's king of the world. He has that ghetto slang going and he's got the my way and take no **** even if your at a professional work place attitude. I generally ignore him and just do my job. He will try and start lots of **** like "what are you looking at?!" or acting like hes the boss. The problem is hes brother with my manager. Everyone I work with agrees he's a prick but they are all too afraid to say anything. Even if we go to our boss and he talks to his brother the problem is now there's bad tension at that point. One time I stopped taking his **** and stood up straight got in his face and told him hes a ******* prick and if he thinks I'm scared of him I'm not. I try and keep my composure and try to stay extremely professional but its getting out of hand. What should I do??