Prince Harry move caused palace 'considerable irritation'