Should I switch?

Are you planning to remain in primary care? If so, the FNP will make you more marketable even if you don't plan to see children specifically. However, you won't have to find a peds rotation and preceptor which I have always read is a challenge in those programs. If you're planning to switch to AGACNP and stay in acute care, that's another option. Job markets appear to be very geographically tied, some areas have a need for FNP/AGACNP and others don't. Good luck with your decision. 

If you want to work in primary care, go for FNP

If you want to work in a hospital, then go for AG Acute Care NP

It sounds like you are talking about AGPCNP rather than FNP?  Acute is another track.  I am similar in that I had years of adult experience.  I found a lot of benefit in knowing my interest early on and gathering work, research, and volunteer experience to support my career plan.  You meet people along the way, patient populations are super important as you are spending full-time RN jobs learning this...NP roles were intended for experienced RNs back in the day.  Anyway, related I see your point.  I think the competition now for NPs requires the successful NP to consider career moves with intent and think with a business and collaborative mindset.  Stay the course much like the medical model, as in the medical settings we are thrown into upon graduating we are expected to know what we are doing.  That LOL has NOT changed over the years.  You also need to network, and you network over time and experience in health care, reputation is everything in optimal work environments.