Only sort-of what Huey said. As Beth notes, the thinking is that fiber doesn't spike your blood sugar, so it won't give you the insulin/leptin/hormone-of-the-week issues that help contribute to fat retention.
Whether that's the correct reasoning or not--and scientists are having a field day arguing the whole insulin hypothesis mess--subtracting the fiver seems to work for me (and for the other low carbers I meet online).
As to whether we need the fiber or not: we're omnivores. Maybe meat is superior to veggies, and maybe not, but we survived for millenia by eating both. And if that's what you need to do to survive in the modern world, I say go for it. Veggies--healthy fiber--can really help stretch a food budget!