It is possible the development of farming had an affect on out bodies overall. Perhaps as farming evolved our nutritional choices narrowed greatly and the lack of some nutrients and the anti-nutrients from some vegetable matter caused micro deficiencies that took their toll slowly over time. Humans are opportunists, one just has to look around to see that we are easily habituated to either what we are used to eating or once we have a steady supply of food we lose the inclination to seek other sources of food.

I think that in the sense that farming is also the cause of civilization in general in addition to any nutritional deficiencies. A lot of HG societies seem to exhibit higher levels of homogeneity. The sexual selection pressures in a civilized agricultural community are probably much different than those found in a nomadic HG group. There may not have been a benefit to growing bigger children who would consume more of the family units sometimes scarce resources.

I think it makes a lot of sense. Farming a]forces permanent settlements, which vastly increase filth, diseases and parasites, as well as changing cultures profoundly (one aspect that seems to be constant is promoting the idea of women as property which leads to earlier marriage of girls) and b]provides a grain-based diet that is much less nutritious and high in starch. Which leads to earlier puberty and more-frequent births, causing a population explosion while at the same time decreasing the quality of our offspring from gestation onward. With poorly-nourished mothers, often starting when they are just past puberty to give birth every 1-2 years instead of every 3-5, fetuses have less maternal nutrient stores to draw upon, and babies and children get poorer care, as well as having to survive on sub-optimal grain-based diets, in dirty conditions, and at constant exposure to disease and parasites. While mothers are are at vastly increased risk of birth injury and death, and are often, physically, very run-down... as well as having stunted growth in some cases from child-bearing while they themselves should be still growing...

ETA: I also think the theory that being a hunter-gatherer promotes higher intelligence is sound.