Charles 'less irritated' with Harry than William