I don't think you're insane. The only thing I would advise you on is to not make yourself an outsider. I am a 21 year old student. There have been two 50-60 year old students in my classes that effectively ostracize themselves from all of the younger students because they are constantly interjecting life experiences they had from many years ago. They were quickly labeled as weird this past semester. This would not be a big deal in regular life, but in college friends are welcome when you need someone to study with or if you need to get a copy of notes. But go for it :) I think you can do it.