I haven't been in psych very long, and have already seen a couple people come and go, for the simple reason they switched thinking it would be easy. That is not a good reason to go into psych. Psychiatric nursing uses it's own skill set.
I debated for a while before making the switch. I did almost 5 years in M/S, Tele/ Neuro, etc. I then did ICU for a year. After being out of work with an injury for a couple months, I was back only a couple days and realized I was ready to switch. Not because I wanted something easier. But I didn't enjoy my job anymore. I used to worry about the loss of skills. But now that I'm working in psych I realize I don't care. I don't see myself ever wanting to go back to that type of nursing. I just didn't enjoy it enough to do it for years.
I now enjoy my job, I enjoy learning about mental illness. It took me a year of feeling like I was struggling in ICU to realize that my biggest problem was myself and lack of interest in what I was learning.
So anyways, I think you just need to figure out what interests you, what do you enjoy learning about, and go from there.