How do I fix a gap between architrave and brick wall?

I purchased an older double brick house that has some interior walls of brick (which I like). Some of the architrave moulding is just up against the brick and because the brick is not perfectly straight it leaves a gap – anywhere from almost nothing to about 10mm. I’m concerned about heat loss (it's cold here for half the year) so I’m thinking of filling it with silicon using a chalking gun but not sure how that will work in the places where there isn’t much gap. Is is the best way to fix it?

The house also has a flat roof so there isn’t enough ceiling cavity space to get up into the ceiling to fix it from the top.