Aligning pre-trained language models ensures they follow human instructions reliably to produce helpful and harmless outputs. Supervised fine-tuning and preference optimization are two key approaches for achieving this goal. In this talk, I will introduce two novel algorithms designed to enhance these two stages. First, I introduce LESS, a model- and optimizer-aware algorithm for data selection. LESS leverages a few curated examples to identify instruction-tuning data that fosters specific capabilities in the model. It avoids relying on surface-form cues by framing data selection as an optimization problem, aiming to minimize the loss on a target dataset (e.g., validation). Our experiments show that training on just 5% of the data selected by LESS outperforms training on the full dataset, with the selected data often transferable across different model sizes and families. Next, I will introduce a simple yet effective algorithm for model alignment, SimPO, which utilizes a reference-free reward formulation based on the average likelihood of model responses. Extensive experiments demonstrate that SimPO outperforms existing offline preference optimization methods, such as DPO, across various settings. Notably, the Gemma2-9B model, tuned with SimPO, achieved the highest rank among <10B models on Chatbot Arena, AlpacaEval 2, and WildBench.
Mengzhou Xia is a final-year PhD student in Computer Science at Princeton University, advised by Danqi Chen. Her research focuses on developing algorithms to build effective language models via data-centric approaches and objective designs under an academic budget. She received her master's degree from Carnegie Mellon University, where she worked with Graham Neubig and her bachelor's degree from Fudan University in China. Mengzhou is a recipient of the 2024 Apple Scholars in AI/ML PhD Fellowship and the 2022 Bloomberg Data Science PhD Fellowship. She has also been awarded as a 2024 MIT EECS Rising Star. Throughout her PhD, she has interned at Meta AI, Microsoft Research, and Bloomberg AI.