in the sentence below why are cancel and keep in ing form? is the preposition "by" omitted from the sentence? was it by cancelling?

It's called a participle clause. It's a way of having more than one verb in a sentence with the same subject without repeating the subject for each verb. There are many possible relationships between the two verbs.

She strictly controlled their kids’ time with their father, keeping them apart on holidays.

She strictly controlled their kids’ time with their father, distressing her parents as well as his.

She strictly controlled their kids’ time with their father, following her mother's example.

She strictly controlled their kids’ time with their father, ignoring their disappointment.

Etc.

This is a very common construction.

He left the house, forgetting to lock the door.

The ball hit a tree, knocking some leaves off the boughs.

The dog greeted him, wagging its tail madly.

Hmm! Just speak your thought ... she's a *****!

"Canceling visits" and "keeping them apart on holidays" are adverbial clauses that modify the verb "controlled." The describe HOW she controlled, just as the adverb "strictly" describe how she controlled.

You could also say "by canceling," etc., but the "by" is not necessary.