If you're after testosterone effects, try Maca instead. While it doesn't directly raise levels, it has similar effects.
DIM can help counter the effects of xenoestrogens such as those from things like BPA.
Fennel and caraway both contain phytoestrogens, so you'd want to avoid both.
You didn't say if you have fat or not, but if you do, note that fat can aromatize testosterone into estrogen. DIM can help block that some, but any male who carries some level of fat will not make full use of testosterone (I've got about 10lbs to go myself.) (Aromatase is the name of the enzyme that does this, has nothing to do with scent.) I do both Maca and DIM.
What foods to avoid for high testosterone? 10 Answers