Could a tv show lose so much viewers due to an actor or actress leaving the show to the point the show would have to cancel?

Yes - it's possible.

It has happened and will, no doubt, happen again.

But the character would have to be an extremely popular one, without whom the whole feel of the show changed for the worse.

After a particular incident, the BBC recently sacked presenter Jeremy Clarkson from its internationally popular motoring show Top Gear. The show's co-presenters decided to leave the show and go with Clarkson to another channel, putting the future of the show in jeopardy. The BBC have now managed find a whole new set of high profile presenters, but it remains unclear whether or not Top Gear has a future. It is quite possible that the BBC may cancel the show completely and cut their losses.

It can happen.

But note the experience of the BBC connected with "Top Gear". All three presenters departed, but the BBC has managed to rescue the show (for the moment?) with three new presenters.

It happened with Felicity, and with a few others, yes.