Researchers have identified subtle forms of racism embedded within artificial intelligence (AI) systems, specifically against speakers of African American English (AAE). Despite advancements in filtering overtly racist content from large language models like ChatGPT, biases related to dialects persist.
A study published on August 28 in Nature revealed that while AI systems generate positive associations when describing African Americans generally, they produce negative stereotypes when prompted about speakers of AAE. These stereotypes mirror or even surpass those held during the 1930s.
The research team, which included University of Chicago Assistant Professor Sharese King and scholars from Stanford University and the Allen Institute for AI, found that AI models frequently assigned lower-prestige jobs to AAE speakers and issued harsher convictions in hypothetical criminal cases.
“If we continue to ignore the field of AI as a space where racism can emerge, then we'll continue to perpetuate the stereotypes and the harms against African Americans,” said King.
King studies AAE as a sociolinguist. She explains that this dialect has distinctive grammatical features such as the "habitual be," used to indicate frequent actions. “She be running” means she runs regularly.
Previous research by King demonstrated that speakers using AAE were perceived as more criminal. Other studies have linked dialect bias to housing discrimination and pay disparities. Inspired by these findings, researchers investigated whether AI harbors similar prejudices against different dialects.
In their experiments, researchers fed several large language models sentences in both AAE and standardized American English. The results consistently showed negative stereotypes for AAE speakers. Comparisons with historic studies on ethnic stereotypes revealed that AI-generated associations were more negative than human-held views from past decades.
Further experiments assessed how these biases influenced decision-making in employment and legal contexts. When matching speakers to occupations, AAE speakers were less likely to be associated with high-prestige jobs. In hypothetical criminal trials, conviction rates were higher for AAE speakers compared to those using standardized American English—68.7% versus 62.1%. Additionally, death penalty sentences were more frequently assigned to AAE speakers—27.7% compared to 22.8%.
Preliminary tests indicated similar biases against other dialects like Indian and Appalachian English but none as pronounced as those against AAE.
Researchers caution about the implications of these findings as predictive AI becomes more integrated into various workflows and business operations. They warn that an employer using AI could inadvertently discriminate against potential hires who speak AAE.
“What we have here is a really big problem because we don't have a solution yet,” King noted.
Human monitoring has reduced overt racism in large language models but has not addressed covert biases like dialect prejudice effectively.
“These findings are really a word of caution about how we're considering its use and how it might disproportionately affect one group more negatively than another,” King added.
Citation: Hofmann, V., Kalluri, P.R., Jurafsky, D., et al., "AI generates covertly racist decisions about people based on their dialect." Nature 633 (2024): 147–154.
https://doi.org/10.1038/s41586-024-07856-5
Funding was provided by the German Academic Scholarship Foundation, the Open Phil AI Fellowship, the Hoffman-Yee Research Grants program, and the Stanford Institute for Human-Centered Artificial Intelligence.