My husband and I are both RN's, so I'm familiar with how his coworkers are mostly female.
Taking everything into account, this is a marriage issue. I'm not sure if the above quote means you have no intention of catering to your wife's concerns. If that's what you mean, then I think you're in the wrong. Your wife and her concerns should always come before your coworkers, and even more so because I assume this is happening at the home that belongs to you both.
And, FTR, my husband has repeatedly been the subject of sexual harassment, and I've witnessed it also, from coworkers who supposedly were "innocent".