Harry and Meghan dropped by their longest-serving staffer