I managed them the wrong way so now that I know better.. What I could say is to make sure you openly create boundaries. If you don't, they will keep intruding and it will continue to escalate. Be respectful yet stern and if you need to make a statement and leave, so be it. Boundaries, boundaries, boundaries!
I can relate to the privacy violated, my mother in law would come to our flat with barely any notice, sometimes first thing in the morning when me and my daughter where still in our pjs and my partner still asleep and she’d come in like “why aren’t you dressed”
Why do you spend time with these people if you hate it so much?