nlp-uncertainty-zoo
Repostory containing model implementations for methods for uncertainty quantification in NLP.
I am Dennis Ulmer, PhD student for NLP at ITU.
I am currently a student under Christian Hardmeier at the NLPnorth group @ IT University Copenhagen and Jes Frellsen @ Technical University of Denmark (DTU).
My current research interests include uncertainty estimation methods with application to Natural Language Processing, learning for low-resource languages, compositionality, inductive biases and much more!
My background includes an undergraduate degree in Computational Linguistics from the University of Heidelberg and a master's degree in Artificial Intelligence from the University of Amsterdam, where I wrote
my thesis under Dieuwke Hupkes and Elia Bruni.
Questions or ideas for collaborations? Message me via the options on the left!
🎉 News
- 01.02.23: I am visiting the SARDINE Lab of André Martins at the Instituto de Telecomunicaçōes and the Instituto Superior Técnico in Lisbon!
- 07.11.22: I released a repository with implementation of many NLP models for uncertainty estimation as a Python package!
- 08.10.22: I have two papers accepted at EMNLP 2022 in Abu Dhabi!
- 08.10.22: Our survey paper pre-print on generalization research in NLP has been released! Check also the Genbench project website.
- 29.04.22: Our paper "Experimental Standards for Deep Learning Research: A Natural Language Processing Perspective" won an oustanding paper award at the Machine Learning Evaluation Standards workshop at ICLR 2022!
- 29.04.22: I received an oustanding reviewer award at the Machine Learning Evaluation Standards workshop at ICLR 2022!
🎓 Publications
2022
- Dennis Ulmer, Jes Frellsen, Christian Hardmeier: "Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity" (Findings at EMNLP 2022) [pdf] [experimental code] [model code]
- Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Rob van der Goot, Christian Hardmeier, Barbara Plank: "Experimental Standards for Deep Learning in Natural Language Processing Research" (Findings at EMNLP 2022) [pdf] [resource repo]
- Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Rita Frieske, Ryan Cotterell, Zhijing Jin: "State-of-the-art Generalisation Research in NLP: A Taxonomy and Review" [pdf]
- Dennis Ulmer, Christian Hardmeier, Jes Frellsen: "deep-significance: Easy and Meaningful Signifcance Testing in the Age of Neural Networks" (Machine Learning Evaluation Standards at ICLR 2022) [pdf] [code]
- Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Christian Hardmeier, Barbara Plank: "Experimental Standards for Deep Learning Research: A Natural Language Processing Perspective" (selected as outstanding paper at Machine Learning Evaluation Standards at ICLR 2022) [pdf] [repo]
2021
- Dennis Ulmer: "A Survey on Evidential Deep Learning for Single-Pass Uncertainty Estimation" (pre-print) [pdf]
- Dennis Ulmer, Giovanni Cinà: "Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection" (UAI 2021) [pdf] [code]
2020
- Dennis Ulmer, Lotta Meijerink, Giovanni Cinà: "Trust issues: Uncertainty estimation does not enable reliable OOD detection on medical tabular data" (NeurIPS 2020 ML4H Workshop) [pdf] [code]
2019
- Joris Baan, Jana Leible, Mitja Nikolaus, David Rau, Dennis Ulmer, Tim Baumgärtner, Dieuwke Hupkes, Elia Bruni: "On the realization of compositionality in neural networks" (ACL 2019 BlackboxNLP Workshop) [pdf]
- Dennis Ulmer, Dieuwke Hupkes, Elia Bruni: "Assessing incrementality in sequence-to-sequence models" (ACL 2019 Repl4NLP Workshop) [pdf]
- Dennis Ulmer: "Recoding latent sentence representations - Dynamic gradient-based activation modification in RNNs" (Master Thesis) [pdf] [code]
📑 Blog
🏆 Awards
- Outstanding paper award for Experimental Standards for Deep Learning Research: A Natural Language Processing perspective at Machine Learning Evaluation Standards Workshop at ICLR 2022
- Outstanding reviewer at Machine Learning Evaluation Standards Workshop at ICLR 2022
💻 Highlighted Coding Projects
deep-significance
Easy and Better Significance Testing for Deep Neural Networks.
📇⚡ token2index
A lightweight but powerful library for token indexing, fully tested and compatible with PyTorch and Tensorflow.
OOD Detection For Electronic Health Records
Collection of a multitude of methods used for OOD detection on Electronic Health Records.