I wouldn’t care. But it would depend on wether I wanted to see them too or not 🤷🏻♀️
Otherwise, they can see the baby when they see them (-:
I can only speak from personal experience with my dad and grandad but depending on if any circumstances that lead you to feel the need to protect your child from them or not wanting contact yourself I said let them visit. They will either prove they're not worth any future time or they'll step up and maybe visiting baby will be a bridge to reconnecting
Just be honest and say well you never wanted to see me before, so I'm not going to let you come over just to see the baby.