Who is wrong in this scenario??

Backstory- my in laws have boundary issues, this has been an ongoing problem since we’ve been married (not between us as a couple, more between my husband and his family). From not respecting our rule of not kissing baby or not washing their hands after smoking, to being annoyed that my husband couldn’t spend as much time with them or we couldn’t host them while I was pregnant because I was having a high risk pregnancy and he needed to take care of me on top of his full time job. We always try to move past things after addressing them but they get annoyed and as a whole family get easily offended by any boundary we set and in turn gossip about us and turn each other against us. It’s caused a break in a lot of the relationships because we stick to our boundaries and don’t let them make us change our mind. The latest issues, they turn up at our house unannounced despite knowing we’re busy and then proceed to be offended because my husband said they should have let us know before coming to give us a heads up. This has happened multiple times. My husband has explained nicely that it’s polite to just let us know even if it’s short notice because maybe we’re indecent having some “alone time” or maybe as a couple were arguing etc or that we may be busy with company or simply just not be home. This has turned into him being called disrespectful and again his whole family has started talking about what a bad son he is. Are we crazy or is this just pure gaslighting?? Who is in the wrong?