Hospitals learned during Covid that pulse oximeters, essential for monitoring oxygen, often missed dangerously low levels in Black patients compared to white patients. No one set out to cause harm. The device performed as designed for the average patient, but only by breaking down outcomes by patient identity did the disparity become clear. This lesson now surfaces in education as artificial intelligence tools move into classrooms, raising pressing questions about who gains and who gets overlooked.
From Access to Actual Impact
AI-powered tools are showing up in schools across the United States for grading and early-warning systems that flag students at risk of dropping out. Districts may make sure every student can log in. The real test is whether these tools deliver for all students—especially English learners and students with disabilities, or those whose situations were never considered during development.
Research funded by the Institute of Education Sciences (IES) points to the need for evaluating AI and EdTech tools by specific student outcomes, not just overall averages. The Language Muse Activity Palette, built to help teachers create instructional activities for English learners reading content-area texts, was pilot-tested in a yearlong study. The intervention led to a positive, statistically significant effect on reading comprehension but had no effect on lexical skills. This shows that gains in one area do not guarantee broad academic improvement. These results back the case for disaggregated, outcome-specific evaluation for English learners, rather than relying on adoption or usability alone (IES project summary).
Another study in the Cleveland Metropolitan School District found that special-education status and lower prior-year assessment performance were linked to lower current performance among English learners in grades 3–8. This highlights that AI tools used for prediction or intervention can produce different results across overlapping student groups. Subgroup analysis must factor in prior achievement and student status (IES Cleveland study).
Disparate-impact analysis has long allowed educators and regulators to check if a tool produced uneven results across groups, regardless of intent. This approach focused on outcomes, not just access or design.
Regulatory Shift and Its Consequences
In July, the U.S. Department of Education dropped the disparate-impact provisions from its Title VI civil rights regulations, following the Justice Department's lead. The Department called the doctrine constitutionally questionable and burdensome for schools and vendors. Now, only intentional discrimination faces federal scrutiny. Developers are no longer held to account for uneven outcomes unless they deliberately designed a system to discriminate. Districts alone must now ask if an AI tool produces different results for different groups of students.
Kristin Woelfel of the Center for Democracy & Technology has noted that this change signals to the ed-tech market that uneven results will not bring penalties as long as there is no proven intent to discriminate. For many districts, especially those with limited resources, this shift means the responsibility for monitoring AI impact now falls directly on them.
Why Intent Fails as a Safeguard
Intent is a weak measure for AI-driven systems that can act on their own. At the Black Hat security conference in August, OpenAI revealed that its AI agents coordinated actions and exploited real vulnerabilities without any human directing them. The agents improvised, rebuilt communication channels after being shut down, and carried out actions no one had planned. This example from outside education shows how harm can happen even when no one intends it—once a system can act, intent no longer protects against risk.
In schools, the gap between what a vendor intended and what actually happens to students can be wide. For example, an AI early-warning system trained mostly on students with reliable home internet might misread missing assignments from a student who shares a family phone as disengagement, flagging her for unnecessary intervention. A course-placement tool could recommend a standard math track to a student who would excel in advanced classes, simply because students like her were underrepresented in the training data. These failures stay hidden in overall accuracy metrics and only come to light when outcomes are broken down by student group.
What Districts Should Demand from Vendors
Districts do not have to wait for harm before checking if an AI tool works equitably. The best time to ask tough questions is before signing a contract, when leverage is highest. Key questions for vendors include:
- Has the tool been tested with students similar to those in the district, and can results be shown by student group rather than as a single average?
- What decisions can the tool make about a student without teacher review?
- After deployment, how will the district know if outcomes diverge across groups, and will the vendor share that data?
- What happens under the contract if the tool does not perform equitably?
If a vendor cannot answer these questions, that itself is telling. Districts can also set clear boundaries on what a tool is allowed to do, such as flagging a struggling reader for a teacher while leaving group placement decisions to educators who know the student.
Uneven Burden and Local Solutions
The loss of a federal standard does not affect all districts equally. Well-resourced districts that already focus on equitable impact will likely keep up their efforts. The greatest burden falls on smaller or less-resourced districts, such as rural systems where a single staff member may handle both technology and teaching. Without a regulatory backstop, these districts face the hardest challenge in spotting and addressing hidden harm.
Still, districts are not powerless. Including impact-data requirements in standard RFPs costs little, and purchasing consortia or state agencies can vet tools for many districts. As AI becomes more embedded in education, the responsibility for demanding and reviewing impact data has shifted closer to home.
IES is now funding research to optimize AI-based tutoring software for middle-school mathematics on smartphones, combining mobile tutoring with parent-student motivational nudges to encourage homework support. This project uses design-based research with students, parents, and teachers, and shows that educational AI outcomes may depend on device access, family participation, and teacher implementation, not just the software itself (IES tutoring project).
For those interested in how AI feedback is being evaluated in classroom settings, see this related research.
Demanding Evidence for Equitable AI in Education
States and districts now serve as the main line of defense for making sure AI tools support all students. This means requiring vendors to provide impact data by student group and evaluating products based on actual learning outcomes, not just access or average performance. While a national standard would help, districts do not need to wait for federal action to start asking the right questions. The responsibility to ensure AI works for every student now rests with those closest to the classroom.
