I've worked in specialty clinics: family practice(all ages), derm, and allergy. I've worked in peds inpatient rehab. The clinic jobs are great for their regular hours and holidays off. Clinic jobs offer a predictable day (usually) and lots of interaction with the same kids and parents over time. I liked that aspect of it, and the families of these kids need a lot of support, they will know you by name, so communication skills are essential!
Many times the progress in the kids is slow, with setbacks, and a few genuine heartbreakers. Some nurses prefer a quick turn-around, more technically complex patient population and would find the day-to-day sameness in age and diagnosis frustrating. It was a good fit for me, as I'm very people-oriented.
A big part of it is the doctor him or herself. You literally work as a unit and spend many hours a week in their company. If your personalities mesh, it can be a dream, if not, a nightmare. Anyway, hope I've given you some thoughts to kick around, best wishes whatever you decide!