This is such a personal decision, and can be a very touchy subject! My answer if you NEEDED the money, would be to just put your guilt aside.
That being said, I have 4 kids and we dont need two incomes right now (it would be nice). I have put off nursing school for quite a while and questioned if thats even what I wanted to do. I think putting my own desires on hold has made it that much sweeter for me:) Does that make sense? I almost want it even more than I did 6-8 yrs ago. You also have to ask yourself what you would regret more, not going to school sooner or missing out on time with your kids while they are little. Only you can answer that. Once I made the decision to just enjoy this time I have with them, I felt a huge weight lifted off. Now I am taking it slow, or or two classes at a time and when I do go they will be 8-15. There will be some activities you miss, but I am finding with my ten year olds that there are so many activities its ok to miss some here and there. Plus its only for two years.