Where can I go to climb a mountain in New Jersey, Pennsylvania, or Maryland?

Delaware water gap has some great climbs for a novice.

New Jersey and Maryland are both pretty flat coastal states with no real mountains. Pennslyvania has mountains but they are not steep rocky snow covered ones like you would find in Colorado or California or other places out west. They are simply tall forested hills with trails to the top usually around 3,000'. You don't need any special equipment or training to climb these, just good hiking boots and camping gear as well as basic fitness. Most of the PA mountains are clustered on ridges and in groups so you usually are not climbing a single peak from base to summit. The area around the Grand Canyon of Pennsylvania near Williamsport in north Central PA has some nice ridgeline hikes.

There are some areas a little farther away where you can climb a single peak the way you describe. In upstate New York there are many peaks in Adirondack State Park. Due west of Lake George up there is Hadley Mountain, a steep peak you can climb from base to summit and back in one day. There are wonderful views from the rocky summit. The High Peaks farther north include Mount Marcy and many others that are around 4,000'.

Up in New Hampshire, 6,000' plus Mount Washington in the White Mountains is the highest peak in the northeast US. There are several trails to the top that are very challenging, especially due to the severe weather on the peak, with snow and high winds almost every month of the year.

Monongahela National Park in West Virginia, just past the Maryland Border has some nice mountains. Seneca Rocks near the towns of Thomas and Davis is a tall stone cliff wall that requires rock climbing skills to climb on the front. But there is a trail up the top on the back that you can hike up in a day.

National forests and state parks abound all around where you live

Use a trail finder too find a nice hike and climb near you

http://www.localhikes.com/

http://www.backpacker.com/

http://www.trails.com/

Since you are new to the hobby read up on the skills and gear you will need

http://www.rei.com/learn/expert-advice.html