We would like to build a language interface for a computer system that has convincing advantages over GUIs and programming. In this work, we select data visualization as a promising application because the action space is too complex for GUIs, yet does not require a full-featured programming language. Commands for manipulating plots often involve potential ambiguities and can benefit from probabilistic modeling e.g. "make the label font size 15", as well as the ability to explore multiple candidates. In this talk, I will describe/demonstrate the challenges and our solutions for designing the action space, collecting data, training models, communicating the action space to users, and methods for interactive learning.
Sida Wang is a research instructor at Princeton/IAS and a graduate of the Stanford NLP group.