DON'T SAY YES!!
You cannot have your cake, and eat it, too!! You work for an agency, the family hired the agency. That is the way it is. Basically all negotiations start with a contract. Look at the one the parents signed. It probably says that they can only deal directly with the agency for their needs for care for your patient.
I have worked with client I got from the agency where I worked, AFTER THE AGENCY WAS NO LONGER INVOLVED, 2 years after her care from them had stopped, and I wsn't working for the agency any longer. Then she was free, as was I to take care of her on my own, as a professional registered nurse.
Of course, her doctor formerly wrote that nursing services are needed and write all the meds, and care you should provide, in circumstances such as I described. You have to keep notes as before, and the parents need to sign legally that they want you to take care of their child, and exactly how many hours, for how much money
Even though the care you'd be giving while the parents went out, seems like babysitting - it really isn't. It's still respite care, which is what they signed with your agency to have......