I too eat beef or lamb liver once a week usually with a side of sweet potatoes drowned in Kerrygold butter. It is indeed delicious. So if there's a reason we should not be eating these foods together, to that I say: ignorance is bliss.
I wouldn't worry much about oxalate issue since a Paleo diet covers so many GOOD things that a few things like sweet potatoes shouldn't be a problem. Nothing in life is perfect and it shouldn't be. Live is death and death is life. Enjoy the meal, you can never use enough good butter and enjoy a Paleo life and stop worrying about everything.