I'd recommend going to the doctor as well. Chills and fever definitely could indicate mastitis and you don't want to leave that untreated.
I went through this and it was mastitis. Antibiotics cleared the fever and body aches within 12 hours! Without meds, it can worsen to an abscess that needs to be surgically removed.
Ive got a doctor's appointment booked for tomorrow, thanks everyone!
Please see a doctor! Chills and fever sounds like mastitis!