Bias in Artificial Intelligence: What Nurse Leaders Need to Know | Knowledge Brush-Up

Lisa Brooks, BSN, MSN, RN

In honor of Martin Luther King, Jr. Day, it seems appropriate to reflect on the conversations about race in the last year. One that stands out in my mind and that may have the biggest impact for years to come is halting the use of facial recognition software.

IBM went the farthest with their decision to stop developing or offering facial recognition software altogether. Microsoft and Amazon paused use by law enforcement to allow lawmakers time to come up with rules.

Why? Because several studies - including one by the federal government published in 2019 - concluded this software shows bias against minorities and women. It misidentifies people of color and women more often than white men, and in some cases was 100 times more likely to produce a false positive.

The implications go beyond facial recognition software itself. As Artificial Intelligence (AI) becomes more common in healthcare, nurse leaders should be asking questions:

Is the Artificial Intelligence we use biased?
How are we introducing bias to AI?
Could AI make existing health disparities even worse?

If you are new to the topic of Artificial Intelligence in healthcare, take a look at my Guide to AI for Nurses.

How Can AI Be Biased?

Most of us think technology is more objective than humans. We know there can be defects that cause strange glitches. But it is hard to imagine technology having racial or gender bias.

Machine learning is a form of AI where computer systems can learn and adapt without instructions. In order to learn the systems must be trained on large sets of data where information is tagged appropriately. For example, if you want to train the system to recognize orange cats, you would give it 1,000 pictures of different types of cats and label which ones are orange.

The application of machine learning to healthcare raises the questions of what these systems are learning and from whom. There are three problem areas where bias can be introduced: (1) lack of inclusiveness in the data used to train AI; (2) bias encoded in the data used to train AI; (3) algorithm errors paired with a lack of human critical thinking.

Lack of Inclusiveness in Data Used to Train AI

One powerful use for AI is in prediction. For example, Netflix uses AI to make better movie recommendations and Amazon uses AI to help you find products you want to purchase faster. Both of these applications of AI use information about you to predict what you may want in the future.

In healthcare, predictive applications could help identify someone likely to develop a health condition, like diabetes, or who will experience side effects of a drug or vaccine before they receive it. But, in order to make predictions, AI has to be trained on those large data sets, which is where bias can occur.

Joy Buolamwini, a researcher at MIT, focused her thesis around a strange interaction she had with the AI in her lab. She was using facial recognition software for a project. But for some reason the software could not tell that Ms. Buolamwini had a face.

When she saw it identify faces on the other people from her team, she realized it may be because she has dark skin. With further research she found the problem was a lack of diversity in data used for training the algorithm.

After testing facial recognition technology from three major software companies, she and her team found error rates of less than 1% for light-skinned men, but over 30% for dark-skinned women. They later discovered the software was trained on a data set that was 77% male and 83% white.

When we extend this issue into healthcare, it is easy to imagine how lack of inclusive data for training AI could make health disparities worse. An algorithm used to identify melanoma trained only only light skin would miss cases on dark skin. It is similar to how women were excluded from cardiac research, and researchers later discovered women present heart attack symptoms differently.

Bias Encoded In Data Used to Train AI

Bias can also enter AI through the way training data is tagged and categorized. For example, clinical notes are widely used in areas like psychology and social work. Natural Language Processing - a form of AI that understands and interprets human language - can extract information from those notes for machine learning.

However, that also means bias baked into those notes can also carry over. If a social worker routinely describes female patients as 'dramatic,' their notes could lead to algorithms that cannot detect anxiety disorders in women.

This is not a far-fetched example. A recent study used Google's cloud image recognition service to evaluate pictures of male and female politicians. It gave men labels like 'official,’ 'white collar worker,’ and 'business person' while it gave women labels like 'smile,’ 'beauty,’ and 'hairstyle.’

This happened because the datasets of labeled photos to train these algorithms already had gender bias, such as showing women cooking and men going to work. The large technology companies are trying to address this problem by being transparent about the bias in their AI instead of correcting the bias. It is basically like a nutrition label on fast-food.

Algorithm Errors and Lack of Human Critical Thinking

One of the big examples of AI bias is the criminal risk assessment instrument used to determine how likely someone is to commit another crime. This algorithm made headlines when a tool used in Wisconsin, New York, California, and Florida labeled African American defendants twice as likely to commit another crime as white defendants.

The criminal risk algorithms take information such as where the defendant lives and employment status to create a score. Judges in states with this tool have used the score to impose harsher sentences.

This resulted in an African American teenage female with no prior offenses receiving a harsher sentence for stealing a bicycle than a middle-aged white male who shoplifted and had several prior arrests. Follow-up done several years later found the man went on to commit armed robbery, while the young woman had no further crimes.

This example highlights how humans cannot afford to turn off their critical thinking while working with technology. Algorithms can produce errors and are not necessarily more objective.

Moving back to the healthcare context, algorithms are not free from error here either. In a study looking at prediction error in psychiatric readmissions at a New England Hospital, the model was found to have a higher error rate for predicting readmission for African-American patients than any other group, and the rate of error for women was much higher than for men.

The bottom line is that nurse leaders need to stay alert, questioning, and cautious. As machine learning becomes increasingly involved in health care decisions, it will be crucial to look at the impact on different demographic groups.

What Can Nurse Leaders Do to Make Sure AI Supports Health Equity?

There are ways in which AI could help to decrease health disparities if channeled in the right direction. To get there, nurse leaders should advocate for guidelines and a shared common goal of eliminating these disparities. At a minimum we will need:

Counter-bias algorithms to test and correct for systemic discrimination

This should be made a basic part of the process prior to an application's approval for use in healthcare.

Greater diversity in data science training and workforce

We are lucky that Joy Buolamwini was in that MIT lab, working with that technology at that time. But what if she was not there? What if she had a different project that did not use facial recognition software? We should not have to rely on luck. Healthcare leaders should push for diversity both at a national level as a requirement for research funding, but also as part of the criteria for AI selection by health systems.

Education of the healthcare workforce that includes how to evaluate algorithm results

We need to do better than the legal system and question the output, especially when it goes against our better judgement.

In Closing

I believe in the promise of what AI can do for humanity. But I also see how important it is for us to understand what tools we are using, who built them, how they were trained, and their impact. To avoid the pitfalls of the legal field, those of us in healthcare must question the technology.

We cannot simply delegate all critical thinking to the algorithm and hope it is right. We are moving into the next great age technology - we must try to leave our biases behind us.

Resources/References

Can AI Help Reduce Disparities in General Medical and Mental Health Care?

Exploring the Potential of Artificial Intelligence to Improve Minority Health and Reduce Health Disparities

Big Data Science: Opportunities and Challenges to Address Minority Health and Health Disparities in the 21st Century

AI Could Worsen Health Disparities