If you go the more traditional route to CNM, where you attend nursing school, receive an RN license, and then continue on to a graduate program in midwifery, most people do choose to work as a RN between the RN program and the CNM program. After you graduate from a RN program (whatever type you may do), you can choose to work only in labor and delivery or a women's services unit and only care for those types of patients.
However, you are, during nursing school (ASN or BSN), going to be caring for all types of patients. Most nursing students spend only one or maybe two clinical experiences in OB, and the rest in other areas of nursing. But let me tell you that the experiences you get in nursing school will absolutely help you when to get into CNM practice. As a CNM, you will still see women who have a variety of medical problems and it helps to have some familiarity with them. For example, I see many women with thyroid issues. I have women with blood clotting disorders in my practice. I see women with migraines, which calls for some experience with neurology. I even diagnosed one of my patients with a seizure disorder after she started having absence seizures during pregnancy. I order and interpret lots of different labwork and imaging/radiology tests. I get all sorts of dermatology questions ("What is this rash?") All these require me to know something about lots of different areas of medicine, not just OB/GYN.
As a CNM, you will be trained to provide primary care for women. Obviously, some practices are more broad in the types of patients you will be required to see than others, but it is to your advantage to not look at caring for patients other than pregnant women (or women in general) as something that you want to avoid. Look at it as a way of gaining experience that may come in handy in the future.
And with regards to the direct entry programs, even in those programs, you will still have the same basic nursing experiences as a student whose goal is a RN license. You will still do med-surg, psych, and other non-OB clinical rotations. That is because, even in a direct entry to CNM program, you will still need to take and pass the NCLEX, and the NCLEX does assume you have some training in all areas of nursing.