Smell and Memory? Is there some kind of link between them?

I always associated the smell of this orange glittry lip gloss with an old house in a town nearby me, and I asked my grandmother recently why I would have ever been in that place, because I didn't know why I've ever been there, even thought I can visualize the inside. She told me that one of her relative used to live there, but the date she gave me that they moved out at would have made me three years old at the oldest, the last time I would have been there. But for some reason, I link this lip gloss smell to it. It's like orange, only it's a sweet orange, but kind of bitter like. It's hard to explain it, but the only time I've ever smelled it before (other than the lip gloss) was once when I went on vacation and I was walking down the city street and I smelled it again, but I don't know where it came from.

And I didn't have that gloss way back then.

What causes it?