There are so many possible answers to this question :) I haven't found bullet-proof evidence that dairy is especially good or bad for us, so I would say to do what you like. The only real issues with dairy seem to reference the protein and sugars, of which butter, cheese, and cream are practically devoid. I eat cheeses, grass-fed butter and a very occasional glass of milk. Kurt Harris, who's blog I've found incredibly insightful, has similar views:

http://www.archevore.com/panu-weblog/2010/1/9/a-taste-of-dairy.html

Edit: My bad, cheese has protein. However, I've read (don't remember the source) that aged cheese alters the proteins in positive ways. Sorry I can't give better support for that. Something about converting A1 to A2?

If you can tolerate dairy, I don't see why you should cut it out. If you want to take a more "middle" approach, is to go for goat dairy instead of cow one. Goat's casein is more tolerable because it's closer to the human milk.

If I were you I'd try going dairy free for a little why, just to see what happens. You might find that you feel better without it, or maybe you'll end up feeling worse, or maybe you won't notice any difference at all. If you're like me you'll find that in some ways your body works better without dairy and in some ways it works worse. No matter what happens though you'll still get interesting and potentially valuable information about your body and what works for it and what doesn't.

I don't believe in an all or nothing mentality... so I suggest keeping dairy in your diet. A cup of greek yogurt and some butter here and there isn't something to worry about. I would NOT however, consume dairy as often as you used to but that's just a personal preference.