nlp-uncertainty-zoo
Repostory containing model implementations for methods for uncertainty quantification in NLP.
I am Dennis Ulmer, Postdoc in Natural Language Processing and Machine Learning at the ILLC at UvA.
I am currently a postdoctoral researcher working with Ivan Titov.
My current research interests include uncertainty estimation methods with application to Natural Language Processing, learning for low-resource languages, compositionality, generalization, and many more!
Before my postdoc, I was a PhD student under Christian Hardmeier at the NLPnorth group @ IT University Copenhagen and Jes Frellsen @ Technical University of Denmark (DTU).
My background also includes an undergraduate degree in Computational Linguistics from the University of Heidelberg and a master's degree in Artificial Intelligence from the University of Amsterdam, where I wrote
my thesis under Dieuwke Hupkes and Elia Bruni.
On other occasions, I also had the pleasure to work with André Martins at the Instituto Superior Técnico, Seong Joon Oh at Parameter Lab, Elman Mansimov at Amazon Web Services and Giovanni Cinà at Pacmed Labs.
Questions or ideas for collaborations? Message me via the options on the left!
🎉 News
- 01.11.24: I started my postdoc position with Ivan Titov at the University of Amsterdam!
- 27.09.24: I successfully defended my dissertation "On Uncertainty in Natural Language Processing" (you can find the thesis on arxiv here)!
- 12.03.24: I will be at EACL 2024 in Malta to present our paper on conformal languag generation and the UncertaiNLP workshop! Say hi 👋
- 20.10.23: We announced our UncertaiNLP workshop: The First Workshop on Uncertainty-Aware NLP, which is co-located with EACL 2024 in Malta!
- 20.10.23: Our paper on generalization research in NLP got accepted at Nature Machine Intelligence!
- 16.10.23: I started yet another research internship at Parameter Lab in Tübingen, Germany!
- 19.06.23: I started my research internship at Amazon Web Services in Seattle!
- 05.04.23: Our paper "Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation" got accepted at the TMLR journal!
- 16.02.23: I gave a presentation at the Helsinki NLP group called "Uncertainty Quantification for Natural Language Processing: Current State & Open Questions". Check out the video and the slides!
- 01.02.23: I am visiting the SARDINE Lab of André Martins at the Instituto de Telecomunicaçōes and the Instituto Superior Técnico in Lisbon!
- 07.11.22: I released a repository with implementation of many NLP models for uncertainty estimation as a Python package!
- 08.10.22: I have two papers accepted at EMNLP 2022 in Abu Dhabi!
- 08.10.22: Our survey paper pre-print on generalization research in NLP has been released! Check also the Genbench project website.
- 29.04.22: Our paper "Experimental Standards for Deep Learning Research: A Natural Language Processing Perspective" won an oustanding paper award at the Machine Learning Evaluation Standards workshop at ICLR 2022!
- 29.04.22: I received an oustanding reviewer award at the Machine Learning Evaluation Standards workshop at ICLR 2022!
🎓 Publications
(* = Equal contribution)
2024
- Dennis Ulmer: On Uncertainty In Natural Language Processing (Doctoral thesis) [pdf]
- Dennis Ulmer, Martin Gubri, Sangdoo Yun, Hwaran Lee, Seong Joon Oh: Calibrating Large Language Models Using Their Generations Only (pre-print) [pdf]
- Martin Gubri, Dennis Ulmer, Sangdoo Yun, Hwaran Lee, Seong Joon Oh: TRAP: Targeted Random Adversarial Prompt Honeypot for Black-Box Identification (pre-print) [pdf]
- Dennis Ulmer, Chrysoula Zerva, André F. T. Martins: Non-Exchangeable Conformal Language Generation with Nearest Neighbors (EACL 2024) [pdf]
- Antonio Farinhas, Chrysoula Zerva, Dennis Ulmer, André F. T. Martins: "Non-Exchangeable Conformal Risk Control" (ICLR 2024) [pdf]
- Dennis Ulmer, Elman Mansimov, Kaixiang Lin, Justin Sun, Xibin Gao, Yi Zhang: "Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk" (pre-print) [pdf]
2023
- Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Rita Frieske, Ryan Cotterell, Zhijing Jin: "State-of-the-art Generalisation Research in NLP: A Taxonomy and Review" (Nature Machine Intelligence) [pdf]
- Antonio Farinhas, Chrysoula Zerva, Dennis Ulmer, André F. T. Martins: "Non-Exchangeable Conformal Risk Control" (Preprint) [pdf]
- Joris Baan*, Nico Daheim*, Evgenia Ilia*, Dennis Ulmer*, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, Wilker Aziz: "Uncertainty in Natural Language Generation: From Theory to Applications" (Preprint) [pdf]
- Dennis Ulmer, Christian Hardmeier, Jes Frellsen: "Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation" (Transactions on Machine Learning Research) [pdf]
2022
- Dennis Ulmer, Jes Frellsen, Christian Hardmeier: "Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity" (Findings at EMNLP 2022) [pdf] [experimental code] [model code]
- Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Rob van der Goot, Christian Hardmeier, Barbara Plank: "Experimental Standards for Deep Learning in Natural Language Processing Research" (Findings at EMNLP 2022) [pdf] [resource repo]
- Dennis Ulmer, Christian Hardmeier, Jes Frellsen: "deep-significance: Easy and Meaningful Signifcance Testing in the Age of Neural Networks" (Machine Learning Evaluation Standards at ICLR 2022) [pdf] [code]
- Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Christian Hardmeier, Barbara Plank: "Experimental Standards for Deep Learning Research: A Natural Language Processing Perspective" (selected as outstanding paper at Machine Learning Evaluation Standards at ICLR 2022) [pdf] [repo]
2021
- Dennis Ulmer, Giovanni Cinà: "Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection" (UAI 2021) [pdf] [code]
2020
- Dennis Ulmer, Lotta Meijerink, Giovanni Cinà: "Trust issues: Uncertainty estimation does not enable reliable OOD detection on medical tabular data" (NeurIPS 2020 ML4H Workshop) [pdf] [code]
2019
- Joris Baan*, Jana Leible*, Mitja Nikolaus*, David Rau*, Dennis Ulmer*, Tim Baumgärtner, Dieuwke Hupkes, Elia Bruni: "On the realization of compositionality in neural networks" (ACL 2019 BlackboxNLP Workshop) [pdf]
- Dennis Ulmer, Dieuwke Hupkes, Elia Bruni: "Assessing incrementality in sequence-to-sequence models" (ACL 2019 Repl4NLP Workshop) [pdf]
- Dennis Ulmer: "Recoding latent sentence representations - Dynamic gradient-based activation modification in RNNs" (Master Thesis) [pdf] [code]
📑 Blog
🏆 Awards
- Outstanding paper award for Experimental Standards for Deep Learning Research: A Natural Language Processing perspective at Machine Learning Evaluation Standards Workshop at ICLR 2022
- Outstanding reviewer at Machine Learning Evaluation Standards Workshop at ICLR 2022
💻 Highlighted Coding Projects
deep-significance
Easy and Better Significance Testing for Deep Neural Networks.
📇⚡ token2index
A lightweight but powerful library for token indexing, fully tested and compatible with PyTorch and Tensorflow.
OOD Detection For Electronic Health Records
Collection of a multitude of methods used for OOD detection on Electronic Health Records.