IMO, you should talk to your manager, and provide rationales for the restrictions you want. Rationales that are about the pt only.
For example, we don't allow visitors during change of shift report because of confidentiality. For the same reason, we ask visitors to step out of the unit when the "Rounds" discussion is about that pt or the pt's next to him. We also ask visitors to not disturb the pt if he is really sick and not getting enough sleep. No visitors during baths and procedures.
It adds up, and although we have 24-hour visiting, we have enough time without the families in our faces that we are able to do our jobs. I think the bits about asking visitors to leave for this or that are not "official" policy in our ICU (I'll have to check), but because it's been done for so long and by the previous managers, etc - our manager supports it.
Asking them to leave can work (no matter what is in your policy) as long as your manager supports you on it. If your manager doesn't.... that's a tough situation. You have my sympathy for having to deal with the families on top of you as you try to do your job. I've heard of places like that in my city, and the nurses hate it!