If veganism and paleo are perceived as "for rich (white) people," that's in part because being able to reject entire commonly-consumed categories of food is a luxury.
It means you have the money to buy other, often more expensive, foods to replace sugar, refined grains, or cheap cuts of meat, and that you have easy access to stores that carry them. It means you have a functional kitchen, with safe, pest-free food storage and a refrigerator that works. It means you have time and energy to cook. It means you have suitable cookware and equipment, or can easily acquire it. It means you already have some cooking knowledge, and know what to do with those fresh ingredients so they don't go to waste. It means that you can choose restaurants based on your dietary preference, rather than what is cheapest. And if you're living with other people and cooking for them, it means you have their support.
I was a vegan in my 20s, when I was also poor. But I'd come from a middle-class family who saw to it that I moved out on my own with cookware, utensils, and food storage containers--so I had a reasonably well-equipped, if cramped, kitchen. I had a good landlord, so that kitchen was also rodent-, bug-, and mold-free; it had a reasonable amount of clean, dry storage space, and all of the (full-sized) appliances worked. I also had a regular kitchen sink.
By comparison, the building across the street was full of efficiency studios with tiny galley kitchens. These had small stoves, a mini fridge, about 3' of counter space, and bar sinks smaller than my big salad bowl (with low water pressure, to boot). If cooking was hard under those circumstances, cleaning up--especially anything fatty--was even worse. The people I knew who lived there generally microwaved everything and used paper plates, or else lived on fast food and snacks, because real cooking was just too damned difficult. Other people I knew, with crappy landlords, went weeks, months--or even permanently--without a working stove, fridge, and/or oven, and lacked the resources to assert their rights as tenants.
I lived in an urban neighborhood that had a lot of Latino and Asian immigrants, so there were produce stands nearby and being a vegan was cheap. I had enough education to know how to use the library (back in those pre-Internet days), where I read books on vegan nutrition and cooking. I already knew how to cook well enough to risk spending money on new grains or vegetables, knowing that I would not end up throwing away unpalatable food nobody would eat.
I didn't have a hungry partner, or children dependent upon their free, crap-laden school lunch, clamoring at me to cook animal foods and refusing to eat vegan options. My home environment was stable--I didn't live with domestic violence, criminal activity, or the constant possibility of eviction or deportation. I lived alone, so I could stock up on food, knowing nobody else would eat it and that I wouldn't be forced to leave and have to abandon it (or all my kitchen stuff) because I didn't have a car. Despite being poor, I managed to juggle the utility bills well enough to keep them all on so I could keep storing and cooking whole, perishable foods.
Among my friends and family, indulging in idiosyncratic diets was almost expected. As a middle-class white female, turning down food and living on plant-based starvation rations was perfectly normal "dieting" behavior anyway, so I didn't catch any real grief for it. There was very little significant social pressure to eat animal foods. And I wasn't dependent upon my friends and family for financial or practical support (i.e., child care, rides to the store), so I could afford to hold firm to my way of eating without fear of losing that support if I pissed anyone off.
And because I knew I would not always be poor--that poverty was just a bump in the road I would one day look back at--it was a lot easier for me to care about the long-term consequences of what I ate. I didn't have any money, but I had shitloads of privilege--enough to assume that of course the future was going to be better. How could it not?
To choose a more health-oriented diet that goes against the norm does not necessarily have to be for "rich people" as far as grocery bills are concerned (though I'm not going to lie and say that paleo is cheap). But it does take a lot of other resources. Education--especially the ability to formulate questions, seek out answers on your own, and evaluate your findings--is a HUGE one. So is social support, domestic "infrastructure," and reliable access to food. And for many people living in entrenched poverty, those resources simply don't exist. That's why it's so easy to dismiss paleo or veganism as part of the realm of "rich white people." They might as well say it's for "people who live on the moon."
Eating off the dollar menu, or living on other cheap, heavily-processed convenience foods is not simply about being too stupid, lazy, or ignorant to do better. For the truly poor and marginalized (which, here in the US, strongly correlates with being non-white in most areas), there are so many other social and material factors that make eating vegan or paleo seem like self-indulgent luxuries for the (white) elite. And even when an individual or family manage to create some success for themselves, make more money, and enjoy a wider array of choices, those food habits formed in poverty tend to die really, really hard. It usually takes at least a couple of generations of middle-class comfort and college education before someone feels they can confidently reject the abundant food they've always been able to take for granted as inferior and unhealthy.
Oh, wow--that's a whole lotta tl;dr, isn't it? Okay, I'll stop now.