How hot should my hydronic (hot water) heating baseboards get?

Sounds like you have found a problem that needs to be investigated. Most baseboard systems are designed for maximum efficiency when the water is 150 to 180 F. You would scald yourself touching pipes that are much beyond 135F. Go to the plumbing store and get a strap-on pipe thermometer (or two) and attach it to various places to see "how far" the heat is getting. The finned-tube baseboard should have a small piece of tubing at each end, where you can attach the thermometer (usually with a spring or clip).

Look for an "air bleeder" valve on each radiator and see if any air comes out when you crack it open. Have a rag under it to catch any water, remembering that it could be wicked hot.

I had plumbers redo five baseboard zones in my house to replace the monoflow tees on the primary loop with straight tees and zone circulators. The radiators went from tepid to sizzling very quickly.

Bottom line, you should NOT be able to touch the radiator pipe when it's supposed to be hot.

Hydronic Baseboards

I also have hot water heat in a 2 story. The fins will not feel very hot. See if you can touch the pipe at the end of the fins, it should be hot if the feed and return lines are hot. The return should feel a little cooler at the boiler. Look in the fins with a flashlight, they may be clogged with dust so you aren't getting heat transfer to the air. If they are very dusty use a vacuum or a brush or whatever you can to clean them off.