how do i make a scary movies "pop out" moments more effective?

There are two factors to a good startle: audio and visual. Make that three if you count environmental distractions.

A lot of home video systems have the audio compressed, so the dynamic range just isn't going to be as extreme. This is different if you are watching it on cable or DVD, where the latter should be better. Even with compression, running the sound through a good stereo or surround system should improve things.

Visually, unless you have a really big TV screen, it probably doesn't fill our field of vision as much as a movie theater screen.

Finally, movie theaters are sealed from external sounds and have carefully controlled lighting. But at home, there are more likely to be lots of distractions, running from minor like the seeing the light on down the hall, to major like the microwave beeping after warming your burrito. All these things have already reduced your concentration on the movie, thereby reducing the STARTLE effect.

You have to create the scary setting for yourself. Watch the movie alone or with just 1 other person..don't have a group around because that will make the movie less scary. Make sure you are watching the movie in a dark place. Set yourself up by a window too, that way while you are watching the movie something outside might catch your eye and it might freak you out depending on what movie it is. Also, the movie doesn't have to be too loud because if you have a pet, for instance, if they are walking around the house and you start hearing floor boards creaking or a door creaking your mind will play tricks on you.