On Scandal, Olivia Pope (Kerry Washington) was President Fitzgerald Grant’s (Tony Goldwyn) most trusted adviser. That’s where the Judy-Olivia comparison ends of course. As all Scandal fans know, Olivia and Fitz were the loves of each other’s lives. And how did Judy break the plotline news to her former boss? “As a good crisis person, you need to quickly reach out to stakeholders — and one of those key stakeholders was the president,” she tells PEOPLE. “I’m like, ‘Okay, I got to tell him first. I got to call him up,’ ” The President, a famous jokester, was amused. Judy sent over talking points for the media, and told him, “I said, ‘If you do not go along with those talking points, I’m going to tell your boss. Don’t play around!’ And he’s like, “You wouldn’t,’ ” she recalls. “And I said, ‘I will call up [your wife] Barbara in a second. I’m not fooling with you.'”