Can I remove an xbox Wireless N Adapter without having to install it all over again?

I wanna remove it after every use because the last time I left it there my brother took it and never gave it back when he took it to his friends house. The I just found it it was with his friend and they had it installed to his system. And I don't want that to ever happen again because I would let him borrow it but when he decides not to give it back, I might risk not ever getting it back. So if I remove it from my console, do I have to restart and re-install the whole thing again?