I worked for 4 years in a busy out/inpatient endo unit in a hospital, and I loved it. Endoscopy is truly an interesting field, and I think you will get out what you put in. There is a lot to learn, and the field changes a lot. I was also able to become a float in the department, after I learned pre-op, post-op and PACU, and gained more valuable skills that way. I do think that if you haven't already, you should definitely research the unit and figure out what they're really all about (i.e.: hours, holidays, etc.). I ended up leaving endo recently for all the reasons you seem to think you'll be leaving behind in the ER: non-stop politics, horrible hours, crappy management, incompetent physicians, etc. The endo unit was supposed to be 4 10-hour shifts, but we were chronically understaffed and had to take call, so I was frequently putting in 14-16 hour days. Also, since the schedule was solely census based, we were staffed daily -- meaning I didn't know what time I was coming to work the next day until the previous afternoon/evening. I could be staffed at 0530, or not until 1200. That kind of work environment was not conducive to a happy life, so I got out. The 3 12-hour shifts without call that I work now are such a welcome change!
As for being "clique-y," you have to learn to work really closely with your co-workers and be part of a team. If one member of the team fails, literally you all fail. You depend on one another to make the unit flow smoothly. I also think endo nursing is very unique, and a type of nursing that you can only understand after you enter the field, so that may also contribute to what people perceive as "clique-y."
I know that not all endo units are as dysfunctional as mine was -- I would just encourage you ask as many questions as possible before you commit! Definitely shadow there also. Good luck to you!