Natural language is a powerful medium to communicate and collaborate with others, often to solve tasks in the world beyond text. While modern NLP systems have pushed the boundaries of language as an interface to instructing models, viewing utterances in the context of interaction with the world and other agents can shed light on new objectives for learning language. People use language to collaborate in far richer ways — not only issuing imperative commands, but also conveying what they know, exchanging information, constructing a shared context together, and more. In this talk, I’ll present our recent work building towards interactive agents like robots and virtual assistants that can use and understand these diverse uses of language in the world. First, I’ll discuss how embodied agents can learn to understand language beyond instructions. I’ll show how language learning can be unified with the view that language helps agents predict the future, enabling them to use language hints, game manuals, environment descriptions, and more to solve a wide range of tasks. Then, I’ll discuss these challenges in the context of modern language model agents, benchmarking their ability to use and understand complex pragmatic language to collaborate with people on hard everyday reasoning problems. Overall, these interactive settings and beyond present new opportunities for language as a means to accomplish things in the world with others.
Jessy Lin is a fourth-year PhD student at UC Berkeley, advised by Dan Klein and Anca Dragan. Her research focuses on building AI agents that use language to collaborate with people and learn from human feedback. Jessy's work has been recognized by an Apple PhD Fellowship and paper awards at NAACL.