I, too, have liability concerns. If you have your own malpractice insurance, I suggest connecting with them to ensure that your coverage extends to such an informal arrangement.
Meanwhile, especially if you don’t have malpractice coverage, I would have a serious come-to-Jesus with the family regarding your respective responsibilities, communication, night-time supervision, etc.
It feels wrong to be so defensive with a friend and neighbor, but if badness happens, a natural reaction in the heat of the moment is to find someone (else) to blame. Unfortunately, in many cases, the outsider comes to mind first. Then, tag - you’re it!