Harry and Meghan don't want to upset the royal family