So although there are people with the obvious lactose and/or casein intolerances/allergies/sensitivities and the possibility that there's something insuligenic, if someone doesn't have any of the previous issues, another likely cause is how it is affecting gut permeability (aka leaky gut).
It's not scientific, but Maria Emmerich is a low-carb nutritional consultant that claims many clients who can't seem to lose weight on LCHF/keto, get past it when eliminating any dairy, including ghee. I think that IF would probably work for those people too, but she doesn't address that one way or the other.
It seems like the deck is stacked against dairy for many people and I'm one where I have to limit it. I don't seem to have the lactose/casein issues, but I definitely seem to have leaky gut issues from it.