Scientifically, many people have shown considerable allergies and gut problems from consumption of milk. Many people do not have sufficient quantities of the enzyme to digest milk after childhood. Historically, milk was probably not a common food item for adults in the paleo, back when we were hunter gatherers. After you were weaned, you probably did not have access to milk. Therefore, it is suspected that many populations of humans have not properly evolved to digest cow milk. Even children sometimes have problems with cow milk and it is suspected by paleos that regularly consumption may cause major or minor gut disruption in many. And from a more modern perspective, most modern cow milk is loaded with pus from sick cows and hormones. It has also been heated and stripped of it's natural fats and then unnaturally blended (homogenized) which may cause additional problems. Some that had problems consuming regular cow milk have done much better on natural raw cow milk like your parents used to get. So some of the problems may come from modern processing as well. Even those populations with a long history of milk consumption did not always drink cow milk and even in those that did, the milk of past generations was much different than modern store bought milk.

Wherease with olive oil, few if any people seem to have any problems consuming it and many studies have shown it to be more healthful that grain oils. Olive oil also has a decent omega 3/6 profile. Obviously, not all populations probalby had access to olive oil in ancient times, but scientifically and metabolically, it seems to hold up well and paleos believe in consumption of healthy natural fats that are not heavily processed. Olive oil can be obtained by relatively natural cold pressing methods without using chemicals, purfumes or dyes to make it palatable, unlike grains oils that are heated to the point of rancidity and then bleached and perfumed to hide their horrible smell.