Oil furnace with a problem?

Oil furnace, hot water runs off of furnace. Furnace replaced last season. Thermostat replaced this season. Same problem with old and new. Set the thermostat to 66 degrees. Furnace comes on but temp in house keeps rising. Does not stop at 66 or even in the 70 range. Must flip the emergency switch to turn off the furnace to get the heat to stop. There is a heat control on one of the lines in the basement which we keep closed in the summer, hot water works fine when this valve is closed. Landlord acts like this is fine, furnace technically works. Any ideas as to what could be causing this is? I know it is the landlord's problem and she should get it fixed but still want to know what could be causing the problem.