Next-word predictions from autoregressive neural language models show remarkable sensitivity to syntax. To what extent can this behavior be explained as an emergent capacity for incremental parsing? I will present work that uses a new class of syntactic probes to 1) find *incomplete* syntactic structures (operationalized as parse states from a stack-based parser) in autoregressive language models and 2) causally intervene on model behavior. The results suggest that implicit incremental syntactic inferences underlie next-word predictions in autoregressive neural language models.
Tiwa is a PhD student in the Department of Brain and Cognitive Sciences and the Computer Science and Artificial Intelligence Lab at MIT, and a student researcher at Google AI. His work uses natural language processing to reverse engineer human linguistic cognition (including comprehension, production, and linguistic reasoning).