So far as I know, that's thought to be the main thing. Of course, the advent of agriculture corresponded to grains replacing other food sources, so the decrease in other nutrient sources could also be a factor.

I think that the grain itself wasn’t as big of a factor in the shortening of the human population as it gets credit for on the Paleo boards. Being taller was an advantage to Grok and his HG buddies because taller people could run faster, jump farther, ect. But when you're chasing down an ear of corn or some wheat speed isn't as important. Thus shorter people who were at a competitive disadvantage were suddenly on a level playing field. Thus their genes were able to propagate more readily into the gene pool. I grew up eating relatively few grains, yet I’m only 5’6. My father was 6’4 at his tallest (he’s been shrinking for several years now) and my mother is 5’0, maybe a little less. Meanwhile my friend Janet grew up eating grains and corn almost exclusively (her parent’s are farmers) yet she stands 6’7 and is the SHORTEST MEMBER OF HER FAMILY (I’m jealous.) I think this is less from our diets and more from our genes. She comes from a tall family; I took after my mother.

Its similar to how poor vision used to be a disadvantage, but now we have glasses to correct for that and thus the genes for bad vision are no longer a disadvantage.

It could also be a factor that farming, especially without the use of modern equipment, is a long, daily, stressful and arduous task. I would imagine the body would spend more time recovering and compensating from even a young age than spending the time increasing body mass and height.

Can't find the blog post but Melissa mentions in one of her post that agriculturist shrank at first but eventually had similar height to HG's, at least that's what I think I read.

Well, I agree that the bigger problem was the displacement of quality, nutrient dense foods with grains that caused most of the reduction in stature. There are tons of humans who lived largely on porridge or bread-like foods for their entire lives.

On the other hand, if we had simply replaced tubers with wheat, the phytate might have interfered with mineral absorption enough that stature could have been affected, even if the same amount of flesh were still consumed.

Perhaps someone has studied height disparities between the peasant and elite classes vs. hunter gatherers. I would wager that the elites didn't take much of a hit in height with the advent of agriculture.