First off, I 100% applaud you for trying to source everything ethically. I try to, as well, and it can get difficult!
Have you looked into finding a farm nearby that will sell you their odd cuts? I'm actually headed down to farm about an hour away later today to stock up on grassfed lamb heart, lamb liver, beef liver and beef spleen for $3/lb, and pastured pig heart and liver for $2.50/lb. The farm is currently low on beef, but I'm a big fan of grassfed beef heart and tongue and where I get it in town it's $3.99/lb, which is still pretty cheap. Tongue and heart taste the most like the regular meat from the animal, so I definitely suggest trying to find a local source for those.
Edit: forgot to mention that you can also go to farmers' markets and specifically request these items from the meat vendors, or better yet call/email the vendors and ask them to bring the weird items to the farmer's market (make sure their price isn't ridiculous first, though). Because they're unpopular and don't sell as well, sometimes vendors don't even bother to bring them to the farmer's markets or at least don't advertise that they sell them, but ask and you shall (usually) receive.
Besides buying bones like Korion suggested, you might also want to look into buying pure gelatin - NOW Foods sells it for extremely cheap if you buy in bulk, but I'm also unsure as to how ethically they source it (their website says it comes from healthy cows, but doesn't everyone say that?). I trust them more than Knox, but I probably should have done more research before buying 5 lbs of their gelatin. Great Lakes is a more trust-worthy source, and still decently cheap.
If you have a Trader Joe's near you, their canned seafood is all wild-caught and BPA-free, and pretty inexpensive. Even if you don't have a Trader Joe's, whatever health-food store you go to probably has stuff like that in stock. TJ's also sells frozen Alaskan cod pieces for $3-something/lb which are pretty good.
Lastly, work on enjoying eggs. If I can do it, anyone can. Eating eggs everyday and loving it is something I thought would NEVER in a million years happen, and within 2 weeks I managed to get over my disgust. It's now normal for me to eat 8 eggs a day, and I frequently go over that number. :)