Can't agree more with the commenters who are encouraging you to be direct. But I would throw in a nod to your part in the dynamic, as in, "I know we've developed a very friendly relationship, but I need us to be just respectfully profesional going forward." I mean, up to this point you have been smiling and chatty and received his attentions. It's important to let him know in so many words there has been a change. When you stop smiling, he just thinks you're having a bad day. It wouldn't be reasonable to ask him to guess by the expression on your face on a certain day how welcome his presence is.
BUT I wouldn't speculate or ask him about what his intentions are, for a lot of reasons. He could say he's just being friendly. I wouldn't use specific behaviors like compliments or whispering - that invites misunderstanding or his going around your preferences with other overly intimate behaviors if he consciously or not doesn't "get it". I wouldn't hide from him, use hints or excuses (you're busy, people are talking, your husband wouldn't like it) or any reason at all, or he might perceive them as objections to be overcome rather than boundaries you are setting.
I wouldn't try to soften it with telling him he's a nice guy, or you're sorry, or if things were different, or anything else that will give him hope or perceived permission to try harder. I might say that it's not personal, but you need your work relationships to be. . . respectfully professional. The bottom line is in order for a relationship to be consensual, both people need to agree on the boundaries, and you could say just that. If he does have actual law enforcement background, once you float the word consensual he will understand very quickly.
I wouldn't escalate this before telling him. As others have said, that invites a he said/she said. I wouldn't escalate once you've told him unless he doesn't change his behavior; your goal I think is to get the unwanted behaviors to stop, not start a harrassment action unless one is needed. I would document the conversation, and any unwanted behaviors that happened after the conversation.
After the conversation, I'd set the professional tone by grteeting him just as you would your other co-workers and going about your day. If he comes to the floor and wants to talk, ask him if there's something he needs, then go back to what you were doing saying politely, "Excuse me, I need to go back to charting," or whatever. With all due respect for the socialization that women get to NOT be direct, to smile and hint and be compliant and wait for someone else to fix it, just talking about this behind his back and making him an object of ridicule is immature and unfair.
These kinds of interactions aren't comfortable for anyone. But look at it from his perspective; he has a wonderful friend at work he thinks there may be a chance there is more with. From his perspective, the flirting has been fun for both of you. You haven't done anything to let him know he's out of line other than showing him a straight face. Do the right thing for yourself, your unit, your management and him, and directly set a boundary.