I think you should go to the interview, be prepared, and be yourself. It's a reality that too many new nurses can be dangerous, there has to be some balance. I would imagine the manager of the 2nd unit you are interviewing for will discuss with the first manager and the HR staff you've interacted with.
It could be a really awesome opportunity. Or it could be something other than what you'd like, which may be workable for the mean time. Get some experience and then do whatever you want (this could also play into the concerns of the first unit).
When I started working, my first job sucked - I'd interviewed with two units, and picked the one I thought would be better for me. The managers competed a little for me, and I should have chose the opposite unit as it was better staffed and less clique-ish, but sometimes that is incredibly difficult to discern in an interview. Doesn't mean you learn/gain nothing though, there is always something to gain somehow/in some way.
I'd take it as positive that the only thing they commented on was being a new grad. That will be fixed with time.