DPP Transformer¶
This module includes an implementation of the transformers using deep determinantal point processes by Shelmanov et al. (2021).
The idea is very related to using Monte Carlo dropout (see nlp_uncertainty_zoo.models.variational_transformer).
The difference is that dropout masks are constructed using correlation kernel between neurons, in order to obtain less correlated predictions overall.
In this model, we implement two versions:
nlp_uncertainty_zoo.models.dpp_transformer.DPPTransformer/nlp_uncertainty_zoo.models.dpp_transformer.DPPTransformerModule: DPPs applied to a transformer trained from scratch. Seenlp_uncertainty_zoo.models.transformerfor more information on how to use the Transformer model & module.
nlp_uncertainty_zoo.models.dpp_transformer.DPPBert/nlp_uncertainty_zoo.models.dpp_transformer.DPPBertModule: DPPs applied to pre-trained and then fine-tuned. Seenlp_uncertainty_zoo.models.bertfor more information on how to use the Bert model & module.