It seems likely that paleo man/woman would have ingested non-nutrient plants for medicine. Possibly for even "fun" during the plentiful times. But generally the idea of regular non-caloric, non-nutrient "foods", high in phytochemicals seems vaguely unpaleo, unless they served some vital phsyiological purpose.
When you look even at ancient cultures use of things like stimulants, they are largely used for endurance for walking long distances (but also religion/spirituality, and sometimes as "mood enhancement")
Be aware, I am talking cooking herbs and spices here too, herbal teas, green and black tea, coffee etc...all things that many people eat or consume daily - not just herbal medicines and plant drugs...
My thought is that alot of these substances are chemically complex, with unknown impact on health.
The body is probably more evolved to deal with small amounts of refined sucrose, than regular hits of hundreds of different relatively novel phytochemicals, especially outside of the context of calories or nutrients.
Are herbs, spices, teas etc really paleo outside of a healing or functional context (ie every day)?