Researchers at Southern Methodist University set out to see how teachers could use ChatGPT to tailor math lessons, but the experiment quickly revealed a major shortfall in AI-driven personalization. In one memorable case, the chatbot suggested using Minecraft to teach circles, radius, and diameter—even though the game’s world is built from squares and doesn’t feature true circles. Minecraft’s popularity among students can make it a lively tool for some math topics, but its block-based design raises questions about its fit for lessons on circular geometry.
Teacher-in-the-Loop: Balancing AI and Human Insight
The study, titled Should There be a Teacher In-the-Loop? A Study of Generative AI-Personalized Tasks Middle School, looked at how seven middle school math teachers used ChatGPT to create curriculum problems tied to students’ out-of-school interests. The students were in seventh grade. Researchers analyzed both how teachers prompted the AI and how students responded, using thematic analysis of open-ended feedback. The main question was whether AI-generated personalization could get specific enough for individual learners. Candace Walkington, Mingyu Feng, Itffini Pruitt-Britton, Theodora Beauchamp, and Andrew Lan found that AI could generate scenarios referencing video games or snacks like Takis, but students responded best when the references matched their own interests, not just broad pop culture. The study drew a line between broad personalization (like mentioning video games in general) and the more precise references students often want, such as a particular game or character (alphaXiv study abstract).
Teachers sometimes turned to AI for help with youth culture, asking for names of popular Twitch streamers or current snack trends. Yet the study found that AI’s suggestions often missed the mark on nuance. Teachers took on the role of evaluating and reshaping the AI’s output, making changes for realism, ethics, curricular fit, and cultural specificity. Professor Candace Walkington said this limitation led the team to consider co-designing generative AI models with students, aiming for references that feel genuine rather than awkward. The study also pointed out a trade-off: broad teacher-facing personalization is easier to scale, but highly specific personalization needs more context and review.
AI in the Classroom: Opportunities and Limits
Personalized learning has been a focus in education research for years. Bringing students’ out-of-school interests into learning tasks—context personalization—has been shown to boost engagement. A 2018 study by Matthew L. Bernacki and Candace Walkington with 150 high-school students found that those who got personalized algebra problems through an intelligent tutoring system reported more situational interest, which predicted accuracy and learning efficiency in the system. The study also found effects on classroom-exam performance and individual interest in math (Journal of Educational Psychology study record).
The SMU generative-AI study builds on this research by exploring how AI can help teachers create tasks that match students’ interests in areas like sports or music, which may boost their interest in learning. However, the study did not offer a statistically powered causal estimate showing that ChatGPT personalization improved math achievement, so no claims about learning gains can be drawn from this study alone.
In the SMU study, teachers used prompting to tweak pop culture references and address issues with depth, ethics, or realism in the problems generated. The model’s value was seen as a tool for assistance and drafting, not as a stand-alone curriculum designer. This helps explain why a suggestion that seems engaging can still fall short if its details or cultural references don’t fit the lesson.
Why Teacher Judgment Remains Essential
Even with AI’s promise, the Southern Methodist University study found that teachers spent considerable time adapting AI-generated content for their students. Teachers’ "prompting moves"—the ways they guided and refined AI output—were central to the process. The findings support keeping teacher oversight, since educators adjusted AI outputs for realism, ethics, curricular fit, and cultural specificity. The study’s design agenda also looks at co-designing generative-AI systems with students, since students may be better at spotting references that feel authentic. There is no evidence yet that student co-designed models have been widely implemented or shown to improve achievement.
Whether the time teachers spend adapting AI suggestions pays off is still an open question. Teachers in the study said the gains in productivity and personalization made the extra work worthwhile. As more research comes in, the broader effects of AI on teaching efficiency and student engagement will become more apparent.
For educators interested in how AI chatbots are already supporting learning in digital environments, a related report explores practical examples and real-world applications.
