I’m sorry but I don’t think this is on you nor is your responsibility to move to him. If you both previously agreed he would stay there with you then he is wrong for going away. Even if he’s fallen out with your mother, for the sake of his child and yourself why wouldn’t he stay and apologise and move on? From what you’ve said it doesn’t seem like a reasonable action to take. It’s not like your mum has done something awful. Perhaps she can apologise too. With the fact they both talk about each other to you, I would just say to them “she’s my mother or he’s my partner and I am not going to tolerate this. I will not be listening to it now on” and whenever they talk bad just tell them to stop and you don’t need to hear it. If they’ve got problems with each other they need to bang their heads together and sort it out so it doesn’t affect yours and your child’s life. If however you both never talked about who’s to live where then that’s a conversation that needs to be had now
If you can’t travel with your son, and trust me I know how difficult it can be and your partner isn’t willing to visit then it depends on what step you want to take next. Whether this is the person you do want to be with when they are being like this. I’m sorry to hear you’re going through that. I think they just need to let bygones be bygones
How old are you? Could you try and find your own place to live. Families are complicated but you need to prioritise your kid and your new family to try and make this work if possible. Long distance with such a small kid must be very difficult but also not sustainable for a family you’re trying to build for your kid.