If it is your calling, and it sounds as if it is, the fact that you cannot get it out of your head, so to speak. Then, no, I would try to put the guilt feelings aside and pursue the dream you have of becoming a nurse. I, too, had the same situation or nearly the same. I didn't necessarily have to go to nursing school...it was a calling. My girls ages were 6 months, 3 years old at the time I entered nursing school. Would I do it again? Yes, in fact, needed to for a self supporting situation since I divorced husband when the girls were still little. Did I have guilt over the decision to go to school? Yes, due to missing out on almost three years of the little ones lives...studying was very hard for me, was not a strong student in high school, didn't know "how" to study, had to reteach myself things that come second nature to a lot of students. BUT, I would not have been able to provide financially as well as I did while my girls were growing up and I wouldn't have been able to support myself NOW .... so...it was a card of fate that not only was I called to the vocation, I ended up needing it to survive, if you will.
Do that what is in your heart. It leads to good things.