What to upgrade to run games on max settings?

ur display is very high for a medium graphic card.

so get ur resolution low or buy a powerful graphic card

You got everything good, except for the video card, HD5770 is not high end, though it is Medium-end, but I can say that the HD5000 series is now somewhat obsolete due to the HD7000 series upcoming release, so I'd suggest you want to wait until the HD7000 series release, else get a HD6870 or an NVIDIA card like a GTX 560 Ti or the current most powerful graphic card - GTX 580. However like I say, the NVIDIA's 600 Series is also set to release sometime this year as well, so you might want to wait a few more month to get it.

And let me tell you one tip, which is look at the number in the name of the graphic card, the higher the number the higher-end/more powerful it will be, for example, HD5800 is more powerful than the HD5700 thus the HD5700 is more powerful than the HD5600 and so on. That's also apply with the NVIDIA's one too. However there's a catch, which is normally the HDx900/ GTX x90 series tend to be a double Video card in one, so be careful before you bought it.

Another way to identify which one is more powerful is by comparing the speed (clockspeed) of the card, same goes with it, the higher the number, the more powerful the card will be.

Definitely a better video card. The 5770 is a decent midrange card, but you're not going to max new games on a decent midrange card.