Many doing a Paleo lifestyle do not have any dairy or only raw grass-fed dairy products. If you know you tolerate dairy then cheese once in a while won't kill you. I don't know what Paleo recipes you're reading that call for Swiss cheese though.
I suggest picking up a Paleo book to read what foods are optimal for a Paleo lifestyle.
On MarksDailyApple.com, there is a post he made about how you should only cut dairy if you're dairy-intolerant, or else it's a wasted effort and you're needlessly depriving yourself of a low carb high protein snack.
Conventional dairy is not that great, but grass-fed dairy is healthier, organic is still OK, and raw dairy is very good as long as you're not intolerant. Cultured dairy, like yogurt, kefir, and cheeses, are better than plain.
In short, dairy is a gray area. Test yourself for intolerances before making a final decision for yourself.
If you follow "paleo" like any other miserable fad, you're not going to stick with it. Think of it like a template to start from, but you can customize it. If you tolerate dairy well there's no reason to restrict it, as aged cheese from pastured animals is one of the more nutrient-dense foods there is.