Billionaire Ken Fisher was barred from an exclusive conference after making sexually inappropriate comments