My boyfriend wants to live with his mother...?

So...my boyfriend and I are talking about moving into an apartment together sometime in the near future.He was talking about one day getting a 2 family house for us and his mom. Not right now but maybe in the future....she is not sick and can perfectly take care of herself. He is just a mammas boy. Was I in the wrong for saying I don't think I would live with him if he did that? We are adults moving out getting our own place and I just really don't want to live right next to his mother. She is really nice and I like her a lot I just don't think I could be happy living with my boyfriend and his mother even if the houses were separated. Now he is upset with me and thinks that I am putting "conditions" on living with him and wants to do it all on his own now in spite even though I said I'm sorry and if that's what you want to do that's what we can do and it really kinda hurts because we were suppose to do this together. But yeah, any advice would be greatly appreciated. thanks :)