Just my $.02 regarding location: I live in New Haven and LOVE it! Of course, I'm from a major urban area in California, so maybe it wasn't such a shift, but I find NH to be a really cool, fun, culturally exciting place to be. It is definitely my favorite place in CT. Great theatre, every kind of restaurant you can imagine, lots of things to do. There are, of course, whole swaths of the city to stay out of, but the downtown/Yale part is beautiful, and East Rock, where I live, is really cute, with a great community and close access to a beautiful park with great views of the Sound. All this is to say that - yes, NH has its seedy areas, but - especially if you're thinking of working in NH, at St. Raph's - it can be a very neat place to live (as long as you're smart about things!).
Re: hospitals - I work at YNHH, not St. Raph's, so I can't say much about it from a work perspective - sorry!
And of course - I'm also speaking as a recent grad, no kids, so my perspective may be very different from yours! Good luck to you though!