Having some grammatical issues with my book, could someone please help?

Re: capitalize and italicize, yes.

Re: Chapter titles, generally words like and, or and the are not capitalized when they appear in book or chapter titles.

The word is er (or alternatively ermm . . . or um . . . or uh . . .)

The word err means to make a mistake so it might confuse readers.

As a suggestion, head to the bookstore and get them to point you to the reference books and pick up a book called the Chicago Manual of Style or a New York Times style manual. If you are writing for a class, ask your teacher which style manual they would prefer to have you use, because what is correct for Chicago or NYT may not work in their idiom of writing. Style manuals are really helpful at answering lots of little questions about consistency like this and are very handy to have. If you write a lot it's a definite must for your tool kit.

You deserve props for being so diligent about writing a good product and paying attention to grammar as well as story. Good luck and I hope the book turns out great.

spells: definately italicized, not sure about caps

chapter headings... maybe a compromise in this using what they call "small caps"??? I don't really care about chapter titles as I don't even look at those when reading a book. I'm ready the damn book!

I like the obvious when you are using the speaking language, so I vote for err and ahh.

You should at least capitalize the first word, and any proper nouns found within the spell (as in Flame of Arnor). Also, you should put quotation marks around the spell as you would for anything else that is said.

As for italics... you probably should if the spell is in another language. Most words in other languages are italicized on principle.

I would italicize the words of the spell, Err means to make a mistake so don't use that one. I would use Uh instead. chapter headings should be in title case, not all caps.

As for the magic words, and italics, I think there is no rule, you can do as you please, but I think italics would be good, and the first word should be capitalised.

I don't think And needs a capital A even in the title.

you want er. Err means to go astray, it's a verb.

Ah and Ahh are ok, but I think Ah means 'I have an idea' and Ahh means 'I am relaxing in the bath'