There has been a slew of work in training helpful conversational agents using Large language models (LLMs). These models draw upon diverse datasets, including open-source repositories, private data, and even synthetic data generated from LLMs like GPT-4. However, curating datasets for supervised fine-tuning involves critical decisions, such as defining task distributions, data volume, prompt length, and more. While prior research underscores the importance of data quality, the nuanced impact of these various dataset factors on model performance remains less clear. In this talk, I’ll present our approach for data curation for supervised fine-tuning and Reinforcement Learning for Human Feedback (RLHF) in the context of training helpful chatbots. Next, I will delve into the results of experiments that illuminate the nuanced effects of different dataset attributes on the training process of helpfulness in chatbots. Finally, I will provide an overview of the current state of chatbot evaluation methodologies and highlight the existing challenges that shape this evolving field.
Nazneen Rajani is a Researcher at Hugging Face working on AI Safety and Alignment using Reinforcement Learning with Human Feedback (RLHF). She is an expert and thought leader in the Large Language Models (LLMs) robustness and evaluation space. Before Hugging Face, she led a team of researchers at Salesforce Research focused on building robust natural language generation systems based on LLMs. Nazneen completed her Ph.D. in Computer Science at UT Austin, focusing on Natural Language Processing (NLP) and the interpretability of deep learning models. She has over 40 papers published at ACL, EMNLP, NAACL, NeurIPS, and ICLR and has her research covered by several media outlets, including the New York Times and Quanta magazine.