Unless your physician is confirming what you know to be true yourself - I'd seek a second opinion.
Just a general obvservation and my own personal opinion - I think physicians words have far too much weight often times. It is incredible how much their advice and medical diagnoses can vary.
Even if you had to work extra-hard through your clinicals and suffer a bit, there are many nursing jobs
that are not in the hospital setting. For example, my mother works at an insurance company with RN case managers. One of the newest RN case managers there just graduated from an ADN program. She has health problems and said that clinicals were brutal for her. But she made it through with the goal in mind of working UR or case management out of the hospital setting. She said that she knew that a nursing degree would open more doors than a social work degree.
Maybe your doctor is right - maybe he is not. No need to reply here, because your health is a personal matter. Just my thoughts.