You and baby will be his family. If. He’s showing willing then maybe accept it and try. Might be worth getting some counselling before baby is here just to help the communication on both parts and help set boundaries and expectations etc.
How about you suggest him come up for a weekend and trail it, then tell him that it is too stressful for you and you need all your energy on yourself and baby. If you don't want to live together you don't have to put up with it just because you're having a baby together, I can understand that it is a difficult conversation to have but you need to do what's best for you and baby
I mean, you're going to have a kid with him, so I think you absolutely have to give it a try!
I think you should give it a try, once the baby comes along you might just appreciate having someone there to lean on or to give you a hand with things, living with someone else is hard but I also think it’s important for your relationship, if you want to stay together and build a family xx