It is a big problem for me because I am lactose intolerant. Causes GI problems.

Going high fat too quickly can cause bowel problems if you aren't adapted to digesting the extra fat. Ramp up slowly to give the liver/gall bladder the ability to process the fat (more bile is required). I'm not saying you need to go high fat, but it works for some of us.

Dairy has lactose and casein, which can both be problematic. Either go off dairy (or stick to ghee) for a while to see if that helps or try to identify the problem. If it's lactose, cream, butter and cheese should be okay. But if it's casein, those are out (especially cheese).

Dairy is not a problem for me and I do eat high fat most of the time.

I can't eat any dairy or I am strapped to the toilet the next day :(. Sigh... but I guess if your constipated there's a solution!!! The best way to see if dairy is problematic for you it to eliminate for a few weeks and re-introduce. Unfortunately, it probably will be, so you will have to make the decision on whether to you will continue eating something you enjoy or improve your gut health. You can always as someone else said keep the ghee and strict dairy kefir or maybe occasional grass-fed dairy if you'd like. I personally had to cut it all out. And high fat that isn't dairy (like Coconut oil) has significantly aided me in my digestive problems.

Eliminate dairy and see what happens. If your GI problems lessen or cease, you'll know dairy was a contributor. After 3 or 4 weeks, you can try adding butter in small amounts back in and, if after a week or two you have no bad reactions, you can try adding cream back in small amounts. Some react to dairy, some don't. You will truly only figure it out by experimenting with your diet.