Kate rejected William's invitation from the queen