Help! narcotic key turnovers

i'm basically new to a nursing home facility and have been working for a week now as a charge nurse on a night shift. am still overwhelmed by amount of things to accomplish throughout the night. what i usually do is to come early and start my chores early. Here is my PROBLEM....the NEXT UNIT always call me to either hold the key for them without counting the narcotics so they can go home early and not wait for the next nurse to come over. in some nights, they will come into my unit knowing that am early and WILL CALL ME TO COUNT WITH THEM and leave the key. my problem is, am not comfortable holding the narcotic key for the other unit. it takes my time just to count with narcotic with them which is not my unit. i come early bec i want to finish my job and not to become as a holder of the narcotic key next door. i am already overwhelmed and still adjusting with my job, counting for another unit which is not mine makes it more overwhelming. The next unit says that it is just but a norm for them to turnover the key to next unit if the incoming nurse at their unit is late. WHAT CAN I DO? am just not comfortable with the idea. HELP!!!