Language models are increasingly being used as interfaces to knowledge. This new paradigm presents exciting new possibilities around synthesizing diverse, multimodal knowledge, but also faces challenges when presented with misleading and uncertain information. In this talk, I will present recent work from my lab on improving the accuracy, efficiency and reliability of LLMs when used for information seeking purposes. I will start by discussing a batch selection technique which improves contrastive learning for multimodal retrieval. Next, I will present methods for dealing with knowledge conflicts which arise when retrieved information conflicts with the model’s parametric knowledge. Then I will present a framework for measuring and improving model calibration when generating long responses consisting of several facts. Lastly, if time permits, I will discuss an extension to speculative decoding which provides a tunable runtime-accuracy trade-off during inference.
Bhuwan Dhingra is an assistant professor of computer science at Duke University and a research scientist at Apple. He has also spent time at Google Deepmind as part of the post-training team for Gemini foundation models. His research focuses on making LLMs more trustworthy and efficient for knowledge intensive tasks. He received his bachelor’s from IIT Kanpur and a PhD from Carnegie Mellon University. His research is supported by grants from NSF, Amazon, P&G and the Learning Engineering Virtual Institute. He was a recipient of the Amazon Research Award in 2021.