Uncertainty metrics¶
This module constains all general-purpose uncertainty metrics used by models in the repository.
An uncertainty metric quantified the uncertainty of a model in its prediction, where higher values of the metric indicate a higher uncertainty.
Uncertainty metrics take the unnormalized logits of a model as an input.
Some metrics might not be included here, since they are model-specified, see for instance nlp_uncertainty_zoo.models.ddu_transformer.DDUMixin.gmm_predict(), which
becomes the log_prob metric for nlp_uncertainty_zoo.models.ddu_transformer.DDUTransformer and nlp_uncertainty_zoo.models.ddu_transformer.DDUBert.
Metric Module Documentation¶
Define some uncertainty metrics for neural discriminators. These usually operate on the predicted logits unless specified otherwise.
- nlp_uncertainty_zoo.utils.metrics.dempster_shafer(logits: FloatTensor) FloatTensor¶
Compute the dempster-shafer metric [2] for a tensor of batch_size x seq_len x output_size.
[2] https://proceedings.neurips.cc/paper/2018/file/a981f2b708044d6fb4a71a1463242520-Paper.pdf
- Parameters:
- logits: torch.FloatTensor
Logits of the current batch.
- Returns:
- torch.FloatTensor
Dempster-shafer metric for the current batch.
- nlp_uncertainty_zoo.utils.metrics.max_prob(logits: FloatTensor) FloatTensor¶
Compute the maximum softmax probability baseline by [1] for a tensor of batch_size x seq_len x output_size. Because we want a high value when uncertainty is high, we actually compute 1 - max. prob.
[1] https://arxiv.org/pdf/1610.02136.pdf
- Parameters:
- logits: torch.FloatTensor
Logits of the current batch.
- Returns:
- torch.FloatTensor
Max. prob. values for the current batch.
- nlp_uncertainty_zoo.utils.metrics.mutual_information(logits: FloatTensor, eps: float = 1e-05) FloatTensor¶
Compute the mutual information as defined in [3] given a number of predictions. Thus, this metric expects a logit tensor of size batch_size x num_predictions x seq_len x output_size.
[3] https://arxiv.org/pdf/1803.08533.pdf
- Parameters:
- logits: torch.FloatTensor
Logits of the current batch.
- Returns:
- torch.FloatTensor
Mutual information for the current batch.
- nlp_uncertainty_zoo.utils.metrics.predictive_entropy(logits: FloatTensor, eps: float = 1e-05) FloatTensor¶
Compute predictive entropy for a tensor of batch_size x seq_len x output_size.
- Parameters:
- logits: torch.FloatTensor
Logits of the current batch.
- Returns:
- torch.FloatTensor
Predictive entropy for the current batch.
- nlp_uncertainty_zoo.utils.metrics.softmax_gap(logits: FloatTensor) FloatTensor¶
Compute softmax gap by [2] for a tensor of batch_size x seq_len x output_size.
[2] https://arxiv.org/pdf/1811.00908.pdf
- Parameters:
- logits: torch.FloatTensor
Logits of the current batch.
- Returns:
- torch.FloatTensor
Softmax gap.
- nlp_uncertainty_zoo.utils.metrics.variance(logits: FloatTensor) FloatTensor¶
Compute the variance in predictions given a number of predictions. Thus, this metric expects a logit tensor of size batch_size x num_predictions x seq_len x output_size.
- Parameters:
- logits: torch.FloatTensor
Logits of the current batch.
- Returns:
- torch.FloatTensor
Variance in predictions for the current batch.