Something similar happened to a coworker of mine, but with an arterial line. As she was removing it, something happened and she accidentally poked the tip of the catheter into her finger. It was probably a quick movement causing the poke and even later in the day she felt a little discomfort in her finger.
Like you, no break in the glove or skin but was still a little worried. Like everyone said, don't sweat it:)