DDU Transformer

This module includes an implementation of the Deep Deterministic Uncertainty (DDU) transformer by Mukhoti et al. (2021). The approach involves fine-tuning a transformer, and then fitting a Gaussian Mixture Model (GMM) on its activations. Model uncertainty is then computed based on the log probability of an input based on its encoding under the GMM.

In this module, we implement two versions:

• nlp_uncertainty_zoo.models.ddu_transformer.DDUTransformer / nlp_uncertainty_zoo.models.ddu_transformer.DDUTransformerModule: DDU applied to a transformer trained from scratch. See nlp_uncertainty_zoo.models.transformer for more information on how to use the Transformer model & module.

• nlp_uncertainty_zoo.models.ddu_transformer.DDUBert / nlp_uncertainty_zoo.models.ddu_transformer.DDUBertModule: DDU applied to pre-trained and then fine-tuned. See nlp_uncertainty_zoo.models.bert for more information on how to use the Bert model & module.

Internally, the Gaussian mixture model is fit in the nlp_uncertainty_zoo.models.ddu_transformer.DDUBert._finetune() and nlp_uncertainty_zoo.models.ddu_transformer.DDUTransformer._finetune() method, which should be called using the data’s validation split.

All the important model info is also encapsulated in the nlp_uncertainty_zoo.models.ddu_transformer.DDUMixin class in order to avoid code redundancies.

DDU Transformer Module Documentation