Well, there is some preliminary evidence that eating enough fat (1, 2), specifically monounsaturated and saturated fat (3) can lead to higher free testosterone levels.

The same seems to be true of eating meat (4, 5, 10), although in at least one of those studies the effect may have been due to the testosterone lowering effect of soy (6). Hey, I guess avoiding soy is pretty paleo though, right?

There is also some evidence that diets high in protein relative to carbohydrates may lower testosterone (3, 7, 8). If you're eating plenty of paleo approved carbs this probably doesn't affect you.

A deficiency of zinc, a mineral found richly in meat, appears well demonstrated to lower testosterone levels (11, 12, 13), an effect which is reversed upon taking in adequate zinc. Oh yeah and zinc absorption appears to be improved by eating paleo; it's increased by meat protein and decreased by phytic acid, casein, soy protein (14).

There's also some evidence from rat studies that a deficiency of vitamin A, that compound commended by liver lovers, can depress testosterone levels (15, 16). Also, a study of male twins found a correlation between vitamin A intake and testosterone levels (17).

I'm sure there are numerous other examples, but these are several notable reasons why a paleo diet may increase testosterone levels.

You may see some temporary (and even permanently) increases in testosterone through several factors. While the SAD is loaded with chemicals stimulating the aromatization of androgens into estrogenic hormones, the paleo diet is very often lacking in such unfortunate properties. In fact, many foods that are consumed on the paleo diet, such as broccoli, animal fats, white button mushrooms, green tea, etc. have anti-aromatase abilities that can prevent androgen hormones from converting to estrogen. The paleo diet is also much lower in phytic acid and other components which may reduce the body's ability to absorb nutrients. Zinc is one example. This mineral is very often depleted in most individuals, while those on the paleo diet have (in theory, I have not actually seen any studies on this subject) higher levels. Zinc is vital in maintaining healthy levels of testosterone production.

However, the body is extremely adaptable, and the symptoms you describe will most likely reverse as soon as your body finds a balance. Also note that testosterone does not necessarily raise aggression. Estrogen's ability to induce aggression and violence is twice as strong as testosterone in studies. Men with higher levels of testosterone are usually much calmer and self-confident, it's only when you raise testosterone levels to very high and unnatural levels that you experience aggressive behavior.

In conclusion, you may experience increased testosterone levels as your body is no longer exposed to aromatase-stimulating phytoestrogens, and is no longer depleted of vital minerals and vitamins such as Zinc and B12 (found in meat and seafood), however, your body will find an optimal balance after a while, and most symptoms will subside.

Could be related to an increase in dietary fats.
I found the figure below from this study; Testosterone and cortisol in relationship to dietary nutrients and resistance exercise