20 pushups isn't actually all that bad for someone who doesn't work out consistently, especially if you were doing a proper pushup. (Though part of this depends on your weight.) People don't often realize how difficult a proper pushup can be.
You say its the same kind of weakness whether you've been lifting something or have your arms raised? Sounds like your muscles aren't used to doing a lot of work. If you have no symptoms other presented, I would be prone to saying you just need to workout. Only then could you tell whether there might be something wrong, e.g. you experience: gains are really slow, you still fatigue quickly after working out for 4-6 weeks. But going from not working out to doing 20 pushups, sounds more than fine. Maybe its an issue of replenishing ATP and your muscles not often worked, don't have many mitochondria to do the workload.
If you are really concerned, get it checked out. But do you feel anxious? Does your heart race? Are thoughts constantly running through your head about whether you are anxious? Do you relax at all? Have trouble sleeping? I would say one symptom is more than likely in your head, rather than a real concern. (Though if it is all in your head, that may be anxiety too.)
If you want some help getting into working out, I would be more than happy helping you develop a quick routine (bodyweight or weighted), pointing you in the direction of one, or I am sure the multitude of people on here would be happy to lend their knowledge.