The rare grapefruit and berries once or twice a week at most.
Personally, if your energy is crashing without it, all the more reason to cut back.
That tells me something is wrong. Blood sugar issues most likely. I would try to find a balance to your blood sugar, try VLC for a while and teach your body to use ketones, once you've adapted, you'll find boundless energy prior to eating and you'll only desire those carbs for glycogen repletion post high intensity workouts.
Then a tuber is a much better choice than fructose.