I enjoyed my drives, too. It gave me time to decompress - not that I needed so much of that with a nicer job. In reality, I got home about the same time as when I lived 15 miles from a more stressful job, where I never got out on time anyway.
I did not mind working weekends and holidays - until I didn't have to anymore! WOW , so that's how the rest of the world lives. Something to be said for that, too. Twice I was snowed into work - had to stay since the next shift didn't get in, but that happened at the closer job, too. Snow days do happen, both getting to work or getting home, but that happens no matter how far away the job is. I would go for the interview - then see what happens.
Worst case, you take the job and it doesn't work out, then you look closer to home next time. Best case - you love it and make lost of $$$$!