Household lithium and/or potassuim compounds that I can throw in the fire (preferably not flammable then lol)?

Unfortunately, putting metal salts into flames usually doesn't make strong colors. It would be easily visible against a blue gas or alcohol flame, but bright yellow flames contain incandescent particles of carbon that emit a continuous spectrum in the flame. This extra light tends to wash out the colors resulting from atomic emissions. In addition, nonvolatile metal ions are difficult to get into flames; simply putting alcohol on a solid salt and igniting it usually doesn't result in much coloration. Suspending some of the salt in the flame, with a wick for example, works better.

That said, copper compounds work very well for turning fireplace flames green and blue. Copper sulfate is sold for this purpose, and sometimes as a fungicide. Copper tends towards green flames unless chlorine is present in the flame. Ammonium chloride is a good source of chloride for producing blue flames; chlorinated plastics like PVC also work, but produce very nasty fumes - not recommended.

Potassium chloride is sold as salt substitute in grocery stores. However, the purple color from potassium salts is pretty weak. Purple fireworks are made by combining a blue emitter such as copper and a red emitter such as strontium.

Lithium batteries are readily available. They contain lithium metal, will burn violently and may explode. These should be used carefully, if at all. Lithium stearate is sold as a high temperature grease in auto and hardware stores. It is probably slightly flammable, which is good, as it helps the lithium get introduced into the flame. Lithium carbonate is the active component of "lithium" pills for mentally-ill patients, and can sometimes be found in pottery supply stores.

The easiest colored fire to make is boric acid in methanol or ethanol. Boric acid is sold in drugstores as an insecticide (it's pretty much nontoxic to humans though); methanol is sold as "gas line antifreeze" in auto stores and gas stations; ethanol can be found as denatured alcohol in the paint stripper section of hardware stores. Methanol is toxic and may cause blindness or death if ingested; denatured ethanol usually has methanol added to it to render it undrinkable (so that it can't be taxed as liquor). Just work in a well-ventilated area, and you'll be fine.

Methanol tends to work better for making a colored flame, as the denaturants in the ethanol can turn the flame yellow. Simply add a good amount of boric acid to the alcohol, stir to dissolve, and ignite some of the alcohol. You should get a green flame.

This works well because the coloring agent, boron, is converted to a volatile compound (trimethyl borate) that enters the flame and burns with the alcohol:

B(OH)3 + 3CH3OH <-----> B(OCH3)3 + 3H2O