I only use them for snacks, mostly almonds/macadamias/brazil, or when I need something in-between lunch & dinner (I usually have about a 7 hour window due to work). However, I only ever have a small handful just to stop the hunger.
I've never found them to be too harmful for me but I don't generally eat more than a small serving & only eat them 3-4times per week. Should note, I eat them raw. I have tried soaking then cooking on low heat to lower the lectins, but didn't really think the small benefit was worth the trouble.
I also use raw tahini (ground sesame seeds) a couple of times a week (1-2 tablespoons in a dish for two people) as it's a good source of calcium and it's great alternative to dairy to make creamy dressings/sauces.
I also find that most people will always have nuts available, so it's a good option when you need to eat something in a pinch. Same with dried fruit - not the best choice nutritionally due to their high fructose content, but good to have if you are caught out without food somewhere.