It does indeed sound like a latex allergy, JellyBean. And most places that I know of today have switched over to latex free because of so many latex allergies in patients and staff. Some of the healing problems on your hands may be from the powder used in the latex gloves but if you break out from having it touched to your face, then I would put my money on the allergy.
Funny, thing is....I am allergic to the non-latex gloves that we use at my hospital. My hands get all itchy and swell and my eyes burn and itch and water. They had to order latex exam gloves for me and I have to be extra careful about checking allergies. There is one brand of non latex that I can use but they are so very expensive that they bought one box for me and I have to put them under lock and key to use for patients with latex sensitivities/allergies. But then, I never laid claim to being normal.....hahahahahaha
Disher is correct in saying that your employer has to make allowance for your allergy as it is considered a disability and they are required by OSHA to provide for adequet personal protective equipment for you that is latex free.
I wish you the best in resolving your allergy.