Find old lace, old stationaries, old music sheet paper, antique items, etc??

Thrift stores are another great and often under-utilized resource. Find the stores closet to older neighborhoods ~ these thrift stores often receive bulk estate donations that are full of ephemera.

If it does not need to have value as a genuine antique, you can make fake ones (ok, antique-looking items) by using tea. Tea-dyed lace is pretty, and you can use any nice lace from a fabric shop. Get one of an older style, of course, and dip it in tea. (Brewed black tea from a teabag or loose tea, still warm.) You can also find sheet music to tea dye. Burn the edges for a fun look, if you like.

The stamps you may be able to find fairly inexpensively. Go to a stamp collectors' shop and ask if they have any old stamps that are not valuable. Tell them what you're doing with them, and perhaps mention that you don't even need them to be in perfect shape. They may have some they can't sell to collectors that they'd be willing to part with fairly easily.

Local antique shop- you can find anything there!

Old magazines for starters... I saw a bunch at the last one I went too... really cool ones like Woman's day or something from the 20's and 30's they were selling for just a few dollars each

Antique stores, estate sales, garage sales,

flea markets,thrift stores or advertise for them.

try ebay, they have just about everything.