Harry suffers blow as pal quits Invictus Games gig