Just as Meg said, it's fine out the fridge below 25° for up to 4 hours. So just pack and go. Enjoy your brunch!
Just to add that if you're going out longer you can pack the milk up in a cool bag with ice packs. Or I used to take it part frozen and just warm it in a cup of hot water to get it nice and warm.
It's fine out of the fridge for 2 hours. We just lob it in the bag and then it's good to go :)