Music coming directly from record player with no speakers?

I just bought a record player and just noticed that there is sound coming from it even though it's not connected to any speakers and it doesn't have any speakers in it. I can hear the music that is on the record and I think it might be coming from the stylus. I know its might be because it's metal and it picks up sound from the vinyl but is it supposed to do this? Cause I could've sworn that when I first put a record on it I didn't hear anything until I connected it to my speakers.