Cleanse skin at night if I don't wear makeup?

Save for very rare situations (big parties, weddings and the like), I only wear mascara and sometimes eye pencil. So no foundation, bronzer, powder, concealer or anything else on my skin. With all fashion mags and beauty tips making a golden rule out of cleansing your skin at night, I just want to ask if it is just as much of a must if one doesn't use makeup. I realise that cleaning is always good for sweat, dust etc, but is it a disaster if sometimes I just crash without doing it, when I am not wearing anything on my skin? Many thanks!