I've been out of school for less than a year, and I feel like it was definitely the first semester due to a number of factors. I was doing one of two pre-reqs per semester, then went to full-time in nursing school, so the volume of work increased dramatically. Also, the first semester contains the basics on which everything else is built--patho, assessment, documentation--so you must know the material backwards and forwards. The material becomes easier in subsequent semesters because you are building on previous knowledge, rather than gaining brand new knowledge without reference.
In later semesters, I just became sick of the work, rather than finding it difficult. Motivation and excitement waned at the end of third semester and the beginning of fourth semester. That was bad too, but for totally different reasons, as you might imagine.