Can you get rid of sensitive skin by moisturizing?

I never had sensitive skin until last year. I use to exfoliate my skin everyday, which eventually became VERY drying, but I dint think anything of it. At the time I dint know much of anything about exfoliating, I just thought it was a brand name soap or something. It always kept my skin super clear, and bright. Until one day, I appeared to have 2 red 'rashes' on both sides of my nose areas. I REALLY would LOVE for this to go away!. Is it possible that if I moisturize my skin, it will go away and be normal again?