Infuriated Charles reaches breaking point with Andrew, sends him into exile early