The fact that the part you find to be the worst, is that people have Confederate flags on their trucks, leads me to believe that you understand that terms such as "slave driver" are generally not racist in the least. As a matter of fact, I feel its a little racist that you find that term troublesome because it came out of a Caucasian persons mouth.. I get told all the time I can use the "male card" can that be taken as sexist? Of course.. As a matter of fact, ive even been told by a coworker "of color" as you put it, that since im white i get the benefit in my job..can that be taken as racism, heck yeah.. Do I care about either? not in the least. Im using as example and personal experience, not comparison, because if I was offended, heck yes I'd leave. So while I do believe that there may be racial slurs happening, I sincerely doubt people are purposely being disrespectful to you due to the color of your skin, if they were and you had any sense of pride at all, you wouldn't even have to ask the question...you'd be out in a heart beat. However I will say, regardless of circumstances, I am very sorry you feel so troubled at your workplace. If you're not comfortable there, then yes, I'd say you should leave.