In many types of human learning, task descriptions are a
central ingredient. They are usually accompanied by a few
examples, but there is very little human learning that is
based on examples only. In contrast, the typical learning
setup for NLP tasks lacks task descriptions and is
supervised with 100s or 1000s of examples. This is even true
for so-called few-shot learning, a term often applied to
scenarios with tens of thousands of "shots".
Inspired by the GPT models, which also exploit task descriptions, we introduce Pattern-Exploiting Training (PET). PET reformulates task descriptions as cloze questions that can be effectively processed by pretrained language models. In contrast to GPT, PET combines task descriptions with supervised learning. We show that PET learns well from as little as ten training examples and outperforms GPT-3 on GLUE even though it has 99.9% fewer parameters.
Hinrich Schütze (PhD 1995, Stanford University) is Professor for Computational Linguistics and director of the Center for Information and Language Processing at the University of Munich (LMU Munich). Before moving to Munich in 2013, he taught at the University of Stuttgart. He worked on natural language processing and information retrieval technology at Xerox PARC, at several Silicon Valley startups and at Google 1995-2004 and 2008/9. He is a coauthor of Foundations of Statistical Natural Language Processing (with Chris Manning) and Introduction to Information Retrieval (with Chris Manning and Prabhakar Raghavan).