We often test language models on what they can say—but what happens when we ask them to think about language instead?
A new study by Gašper Beguš, Maksymilian Dabkowski & Ryan Rhodes (2025, IEEE Transactions on Artificial Intelligence) explores just that. The researchers tested four major language models—GPT-3.5, GPT-4, Meta’s Llama 3.1 & OpenAI’s o1—on a series of linguistic analysis tasks that human linguists perform daily: identifying syntactic ambiguity, drawing tree diagrams, analysing recursion & phonological rules, & explaining sentence structure.
Unlike most AI benchmarks, these tasks didn’t test the models’ use of language, but their metalinguistic ability—their capacity to reason about how language works. Each model was prompted with custom-designed problems, & linguistics postgraduates graded the outputs.
The results were clear: OpenAI’s o1 stood out. It consistently produced more accurate, coherent analyses, generating syntactic trees, recursive structures, & phonological rules that resembled the reasoning of trained linguists. The authors suggest that o1 may be the first model to show genuine metalinguistic ability—not just statistical pattern-matching, but structured analytical reasoning similar to how linguists think.
While there may be no direct implications for classroom practice yet, this research nevertheless invites reflection. It challenges us to reconsider what it means to “understand” a language—& how humans, too, develop the ability to talk about language as a system. For teachers, it’s a reminder of how metalinguistic awareness—recognising patterns, analysing grammar, explaining rules—is both a uniquely human strength &, now, a frontier in AI cognition.
If AI can now analyze language as well as use it, what does that say about the nature of linguistic understanding itself?
Leave a Reply