The royal rumor that hurt Diana the most