Linguistic evaluations of how well LMs generalize to produce or understand novel text often implicitly take for granted that natural languages are generated by symbolic rules. Failures of LMs to obey strict rules have been taken to reveal that LMs do not produce or understand language like humans. I suggest that LMs’ failures to obey symbolic rules may be a feature rather than a bug, because natural languages are not based on rules. New utterances are produced and understood by a combination of flexible interrelated and context-dependent schemata or constructions. We can use constructions to develop new evaluation paradigms for LMs, which focus on rare, non-compositional phenomena, and in doing so define the boundaries of LMs’ linguistic capabilities. I showcase this in LLM evaluation studies on the causal excess and the caused-motion construction. Recognising that LMs are modelling most of these phenomena quite well, we can then move on to studying how they do this, and comparing their behaviour against competing theories of human language learning. As an example for this, I present work comparing LMs’ learning of dative preferences against that of humans, and a study comparing LLMs’ usage of derivational morphology against different cognitive models. I conclude by urging the community to not climb the wrong hill with respect to cognitive plausibility of LMs, and instead focus more on how, not if, they acquire language.
Leonie Weissweiler is a postdoc at UT Austin Linguistics, working with Kyle Mahowald, funded by a fellowship from the German Research Foundation (DFG). She completed her PhD at LMU Munich with Hinrich Schütze, on Computational Approaches to Construction Grammar and Morphology. She previously completed her B.Sc. and M.Sc. degrees in Computational Linguistics and Computer Science at LMU Munich and Cambridge University. Her work has been supported by fellowships from the German Academic Exchange Service and the German Academic Scholarship Foundation. Her research centres around applying language models to the study of language, particularly in the areas of construction grammar, morphology, and typology, as well as the evaluation of language models using connectionist accounts of language.