This talk is part of the NLP Seminar Series.

Natural Language to Code Generation for Interactive Data Science

Pengcheng Yin, Google
Date: 11:00am - 12:00pm, April 6th 2023
Venue: Room 287, Gates Computer Science Building; Zoom (link hidden)

Please note that this talk is NOT open to the public.

Abstract

AI pair programmers, powered by large language models (LLMs) trained on code, have significatly enhanced developers' producitivity by automatically synthesizing programs based on their natural language (NL) intents. Recent studies suggest that developers prompt code LLMs using succint natural language comments in multiple turns, where each turn depends on the programmatic context and interaction history of prior turns. To evaluate the performance of AI pair programmers in such multi-turn interations with developers, we build ARCADE, a benchmark of 1082 code generation problems using the pandas data analysis framework in data science notebooks. These interactive computing environments are ubiquitous among data scientists to perform data wrangling and analytical tasks. ARCADE features multiple rounds of NL-to-code problems from the same notebook. It requires a model to understand rich multi-modal contexts, such as existing notebook cells and their execution states as well as previous turns of interaction. To establish a strong baseline on this challenging task, we develop PaChiNCo, a 62B code LLM for Python computational notebooks, which significantly outperforms public code LMs. Finally, we explore few-shot prompting strategies to elicit better code with step-by-step decomposition and NL explanation, showing the potential to improve the diversity and explainability of model predictions.

Bio

Pengcheng Yin is a Research Scientist at Google Research, Brain Team. He received his Ph.D. degree in language and information technologies from Carnegie Mellon University. His research interest is building natural language interfaces that allow humans to communicate with computers in their own language. Most of his work focuses on semantic parsing and code generation --- synthesizing machine-executable programs from natural language utterances.