If you think it will ultimately be more rewarding, personally, then I say go for it. Just two caveats:
1) Good for you for being proactive and doing some shadowing before you commit to starting a career change. However, unless you are definitely specifically interested in the ED, I would suggest that you try to get some time in on a med-surg floor; it's a whole different animal from ED nursing, and it's where you'll more likely end up for at least part of your career.
2) Be sure not to burn any bridges at the place where you're working now, especially if it's a good job. Even though you wouldn't be looking to enter the nursing workforce for at least a couple of years yet (and who knows where the economy will be then), be advised that most new grads right now are having a really hard time landing that first job, and it would be good if you could hang on to your current job as a backup.