Natural language understanding is a quickly growing field, both in terms of modeling and data work. However, there is little agreement on what specific reasoning "skills" the models are supposed to learn, which is why many resource descriptions and error analysis sections rely on ad-hoc categories. This talk presents a taxonomy of verbal reasoning skills for future resource and model analysis work, based on the findings of a large-scale survey of current resources for reading comprehension and question answering. I will also discuss the key challenges in model evaluation and collecting data for training/testing specific "skills", as well as some proposed solutions.
Anna Rogers is a post-doctoral associate at the University of Copenhagen. Her main research areas are analysis of deep learning models for NLP and design of training/evaluation data. She is also active in the sphere of NLP methodology, co-authoring the reviewer tutorial at ACL Rolling Review, contributing to the ARR Responsible NLP checklist, and organizing the workshop on Insights from Negative Results in NLP.