We've been considering getting pigs. My husband wants to get six - he's a "Go big or go home" type. Someone told him "You should start smaller! That's a lot of work!" His response was that, if it's too much work, we'll just have a barbecue.
We've had goats for about a year now, with no previous farming experience. That was a similar situation. We got 7 goats and 25 chicks within a two week period. It hasn't been particularly difficult. If you're reading websites about goats, it makes it sound like they'll dissolve into piles of bloody poop the minute you bring them home unless you buy a vet's office and monitor them 24/7, but we've had no problems. I've seen a few other people get started with a similar lack of issues. They're adorable, especially as babies, and very friendly if you work at socializing them. Personally, I find them much more fun than chickens, though I suppose chickens are more cost-effective.
They do eat a lot of hay, especially if they're pregnant or nursing (obviously necessary for dairy) which is a major consideration unless you live somewhere with adequate pasture year-round.
If you want meat, you have to be willing to kill (or have someone else kill) an animal you've raised. If you have someone else do the butchering, that adds to the cost of the meat. If you decide you don't have the heart for that and just want dairy, you need a plan to deal with the babies that are a necessary part of that process.
Vacations are certainly something that has to be planned for, but the same is true of any pet (our big dogs are at least as much an issue as the goats in that regard), and it isn't an insurmountable hurdle. We have a housemate, so we can alternate vacation times. My friend with goats boards hers with the person she bought them from. Friendly neighbors or responsible teenagers wanting to earn some extra money are a possibility also, especially if you're just talking 1 or 2 goats.
So anyways, yes, look at the downsides. But always keep in mind that this is something people DO, and the hurdles aren't insurmountable.