Raw (unpasteurized) milk is not particularly dangerous IF the source of the milk is a dairy that takes their responsibilities seriously, as nearly all do. In the past, the most dangerous potential pathogen was tuberculosis bacilli and because of that, all of the 22 states that permit raw milk sales have strict requirements for tuberculin testing of herds. These days, most of the problems come from salmonella and listeria but as you know, these pathogens are responsible for a lot of food poisoning generally, not just from dairy products. Very, very few of these incidents are from raw milk.
While pasteurization is reasonably effective, it is far from fully effective and there are many more incidents of food poisoning traced to pasteurized milk than to raw milk. Of course, this is primarily because the numbers of raw milk drinkers are small compared to pasteurized milk drinkers but the point is that each has some level of risk. One of the arguments for raw milk is that it forces farmers to pay close attention to their herds in everything from food to hygiene. Milch cows producing for raw milk farmers get grass and silage, are minimally confined, are kept in much cleaner housing and are tested regularly. This is not always the case (sadly, it's rarely the case) for cows from other farms.
I'm personally not sold on the concept that raw milk is more beneficial than pasteurized (though milk from grass-fed cows is healthier, even if it's pasteurized) but I can state unequivocally that it tastes much better. Most pasteurized milk is a pale imitation of what milk really tastes like. When we were kids, my dad, who was as far from a health food nut as you can get, would occasionally take us to visit the dairy farm that delivered to our house (these were the days of milkmen, glass bottles and unhomogenized milk). This farm milked all their cows by hand because they believed that the cows were more productive and lived longer by doing this and they would also allow visitors to milk the cows. The real treat was when they would take milk directly from the stainless milk pail and give us a taste. There's really nothing like it. We always wound up buying a couple of bottles of raw milk to take home (the milk they delivered was pasteurized).
If you are still not ready to take the plunge and try raw milk, you can get closer to the taste by finding a dairy with a grass-fed herd that uses the low-temperature pasteurization process (most dairies use the high-temp process because it takes less time). While any pasteurization process effects the taste of milk significantly, the low temp process is less destructive to some of the flavor components and results in a much, much better tasting glass of milk. You'll pay for the extra flavor and health benefits - milk from grass-fed cows is much higher in linoleic fatty acids - and is worth the extra cost (it's about double the usual stuff).
In short, though I don't regularly drink raw milk, there's really no good reason for a reasonably healthy person to shy away from it.