Water and high fibre foods will help. Weetabix is great, and prune pouches too.
We’ve had the same problem, offer as much water as possible and lots of whole grain/fruit and veg in their meals. I’ve started offering some fruit/fruit pouch with every meal, makes a huge difference.
My baby had the same thing, it's so painful 😣 I feel you! I had to take her to the GP cause she was constipated. They prescribed me a medicine and asked me to give her a lot of water. It's possible dehydration cause constipation as you said your baby not having milk. Water is so important! All the best xx