Should I be worried that my workmen weren't wearing a mask?

The risk is probably minimal. 

There is new evidence that the virus can be transmitted in smaller "aerosol" droplets which can linger in the air for a while.  But most of it seems to be transmitted by larger droplets which will fall to the floor fairly quickly after being exhaled.   The big danger is from being near someone who is infected and breathing out.  The fact that these guys didn't wear masks isn't good, and you may want to complain to the company about that.  But if you stayed out of the room when they were there then you have reduced, if not eliminated, your risk of infection.  By staying out of the room they were in, you also reduce the risk of being infected.  That's because you give the particles more of a chance to settle and also more of a chance for environmental factors, such as sunlight, to kill the virus.  Opening the windows also helps since you're replacing air which may contain the virus with air which does not. 

If you stayed away from them, you shouldn't have much chance of getting covid 19.

The landlord should have asked them to wear a mask. They should when they're indoors with people that they don't know. You should have a mask in your house, in case a stranger has to come in it. 

You have the right to refute entry to anyone who isn't wearing a mask. Call the company they came from and let them know. Tell anyone to wear a mask when they come into the house, and that includes the police, or refuse entry.