Meghan warned it's 'too dangerous' to visit her father