drink water immediately and it will help within minutes, literally. but that's a bandaid to an sympton ailment that you are feeling because your kidneys are complaining.
personally, i've had several kidney stones before going paleo. i know exactly what that feeling is like (waking up with kidney pain). my stones were always calcium oxalate. chris masterjohn makes an extraordinarily strong case on the WAPF website about the role of fat soluable vitamins A,D and K playing a huge role in how your body distributes and absorbs calcium. so basically the fermented cod liver oil and high vitamin butter oil blend could play a major role with kidney function and calcium.
for the last 6 months, I've been taking that combo and I have noticed a huge improvement. now, i also completely changed my diet to lacto-paleo, but I pretty much never have kidney pain anymore and I haven't had a stone since. also, ironically, since switching to paleo, i drink much less water and yet still no kidney pain, almost like there's nothing there is no longer any calcium oxalate to 'flush' anymore.
hope that helps,