The 24-year-old posted to Instagram in April in honour of Braff's 45th birthday, showing him and their foster dog.
Within minutes, she had to "turn off the comments" on the post due to the incessant "hateful comments" she received.
People had very strong opinions about her relationship, mostly because of the 21-year age gap between the couple.
Appearing on the Sue Perkins: An hour or so with... podcast this week, Pugh spoke about the reaction.
"I've always found it funny, how I can be good enough for people to watch my work and support my work and pay for tickets, and I'm old enough to be an adult and pay taxes, but I'm not old enough to know who I should and should not have sex with?" Pugh told host Perkins.
"Yet again, once again, it's making a young woman feel like sh*t for no reason."
Pugh further defended her relationship in an interview with Elle UK in May.
"I have the right to hang out and be with and go out with anyone I want to," she said, saying that people had no right to "educate" her on her private life.
"I know that part of being in the spotlight is that people might invade your privacy and have opinions on it, but it's bizarre that normal folk are allowed to display such hate and opinions on a part of my life that I'm not putting out there.
"It's a strange side of fame that you’re allowed to be torn apart by thousands of people even though you didn't put that piece of you out there. My point to all this is that isn't it odd that a stranger can totally tear apart someone's relationship and it's allowed?"