Recent advances in large auto-regressive language models have demonstrated strong results in generating natural languages and significantly improved the performances for applications such as machine translation and summarization. However, when the generation tasks are open-ended and the content is under-specified, existing techniques struggle to generalize to new scenarios and to generate long-term coherent and creative contents. This happens because the generation models are trained to capture local surface patterns (i.e. sequences of words) following the left-to-right order, and it is hard to impose structural or content control/contraints for the model. In this talk, I will present our recent works on controllable text generation that go beyond the prevalent auto-regressive formulation. We propose a novel insertion-based generation model and a controllable decoding-time algorithm to steer models to better conform to constraints, with applications to poetry generation and machine translation.
Nanyun (Violet) Peng is an Assistant Professor of Computer Science at the University of California, Los Angeles. She received her Ph.D. in Computer Science from Johns Hopkins University, Center for Language and Speech Processing. Her research focuses on the generalizability of NLP models, with applications to creative language generation, low-resource information extraction, and zero-shot cross-lingual transfer. Her works have won the Outstanding Paper Award at NAACL 2022, the Best Paper Award at AAAI 2022 Deep Learning on Graphs workshop, and have featured at the IJCAI 2022 early career spotlight. Her research have been supported by several DARPA, IARPA, NIH grants and industrial faculty awards.