Introduction

Part 2 of the uncertainty series · Read Part 1: the foundations

Uncertainty estimation provides a structured way to measure how confident a deep learning model is in its predictions. These techniques play an essential role in handling noisy inputs, supporting decision-making in safety-critical settings, and identifying out-of-distribution (OOD) samples. Broadly, existing methods fall into sampling-based and sampling-free categories. Each class comes with its own advantages and limitations in terms of accuracy, computational overhead, and scalability—topics discussed in the following sections.

As discussed earlier, many uncertainty estimation techniques originate from probabilistic principles, yet most ultimately focus on producing a scalar quantity that reflects confidence in a prediction rather than modelling the full predictive distribution, which is often computationally prohibitive. In a simple 1D regression setting, for example, the model may output both a mean and an associated scale parameter such as a standard deviation [15]. Different approaches vary in how these quantities are obtained. The following sections outline how commonly used methods compute uncertainty and summarize them in a high-level comparison table (Table 1). Before introducing these methods, we first establish a more formal view of uncertainty from a Bayesian perspective.

Methods at a Glance

The table compares broad computational tradeoffs. Costs are approximate and depend on the architecture, implementation, and number of predictions; use the linked papers for method-specific details.

Table 1. Uncertainty-estimation methods and their approximate costs relative to a single model.
MethodTrainingInferenceMemoryEasy integrationRegressionClassification
Sampling and multiple-prediction methods
Deep Ensembles [1]N×N×N×✓✓✓
MC-Dropout [2]1×N×1×✓✓✓
Snapshot Ensembles [3]1×N×N×✓✓✓
FGE [4]1×N×N×✓✓✓
Bayes by Backprop [5]1×N×2×✓✓✓
BatchEnsemble [6]1×N×1×✓✓✓
Masksembles [7]1×N×1×✓✓✓
Packed-Ensembles [8]1×1×1××✓✓
MIMO [9]1×1×1××✓✓
Iterative Uncertainty [10]1×N×1×✓✓✓
ABNN [11]1×N×1×✓✓✓
Prediction Depth [12]1×1×1×✓✓✓
Subnetwork Inference [13]1×N×1×✓✓✓
BatchNorm Uncertainty [14]1×N×1×✓✓✓
Single-model and sampling-free methods
Single Model [15]1×1×1×✓✓✓
SNGP [16]1×1×1×✓×✓
DUQ [17]1×1×1×××✓
DDU [18]1×1×1×××✓
ZigZag [19]1×2×1×✓✓✓
Experts Don’t Cheat [20]1×2×1×✓×✓
Evidential Learning [21]1×1×1×✓✓✓
Prior Networks [22]1×1×1××✓✓
Posterior Networks [23]1×1×1××✓✓
Variance Propagation [24]1×1×1××✓✓
Orthonormal Certificates [25]1×1×1×✓✓✓
Conformal Predictions [26]1×1×1×✓✓✓
How to read the comparison

N is the number of predictions or ensemble members. Training, inference, and memory costs are rough multiples of a single-model baseline, rounded down unless substantially larger. The original comparison assumes sequential evaluation and model weights residing in GPU memory; batching and hardware can change the practical cost.

“Easy integration” is a subjective indication of how much an approach changes the architecture, training procedure, or required data. Checkmarks indicate the capabilities described by the cited method, rather than a guarantee for every variant. Conformal coverage additionally depends on assumptions about calibration and deployment data.

The Bayesian View

From Priors to Posteriors

Bayesian methods offer a principled way to incorporate uncertainty into machine learning models by treating model parameters as random variables rather than fixed quantities [2, 5]. The central idea follows Bayes’ theorem, which connects the posterior distribution of the parameters \(\theta\) given observed data \(D = \{(x_i, y_i)\}_{i=1}^N\) to the prior distribution \(P(\theta)\) and the likelihood:

\[P(\theta \mid D) = \frac{P(\theta) P(D \mid \theta)}{P(D)} = \frac{P(\theta) \prod_{i=1}^N P(y_i \mid x_i, \theta)}{\int P(D \mid \theta) P(\theta) \, d\theta}\]

where \(P(\theta \mid D)\) is the posterior distribution, representing the updated belief about the parameters after observing the data, \(P(\theta)\) is the prior, capturing prior knowledge or assumptions about \(\theta\), \(P(D \mid \theta) = \prod_{i=1}^N P(y_i \mid x_i, \theta)\) (assuming the data points are independent and identically distributed) is the likelihood, describing how probable the observed data \(D\) is given the parameters, and \(P(D)\) is the evidence, a normalizing constant ensuring that the posterior is a valid probability distribution. From this formalization, many familiar loss functions and regularization techniques in machine learning can be derived by specifying assumptions about the likelihood \(P(y_i \mid x_i, \theta)\) and the prior \(P(\theta)\). These assumptions determine the model’s behavior and the constraints imposed during training, providing a unified Bayesian perspective on various learning approaches.

The Posterior Predictive Distribution

In the Bayesian approach, predictions for a new input \(x\) are made by marginalizing over the posterior distribution of the parameters. The predictive distribution is expressed as:

\[P(y \mid x, D) = \int \color{#2563eb}{ \underbrace{P(y \mid x, \theta)}_{\text{Aleatoric}} \, } \color{#16a34a}{ \underbrace{P(\theta \mid D)}_{\text{Epistemic}} } \, d\theta\qquad(1)\]

This equation highlights the two main sources of uncertainty in Bayesian modeling. The term \(P(y \mid x, \theta)\) represents aleatoric uncertainty, which arises from inherent noise or ambiguity in the data. It reflects the variability in predictions for a fixed model parameter \(\theta\) and is irreducible, even with additional data. In contrast, \(P(\theta \mid D)\) captures epistemic uncertainty, which stems from the model’s lack of knowledge about the true parameter values due to limited or insufficient training data. This uncertainty can be reduced by gathering more diverse or representative data, leading to a more confident posterior.

Why We Need Approximations

However, directly computing \(P(\theta \mid D)\) is intractable for complex models like deep neural networks due to the high-dimensional parameter space and the integral required to compute \(P(D)\). To address this, most existing uncertainty estimation methods for deep learning attempt to implicitly or explicitly approximate the Bayesian integral to make it computationally feasible. A classical strategy is to replace analytical computation with sampling. In this context, sampling from \(P(\theta \mid D)\) provides an estimate of the integral, and we refer to approaches based on this idea as sampling-based methods.

Sampling-Based Methods

Approximating the Predictive Distribution

Sampling-based methods derive uncertainty estimates by generating multiple predictions from the model and analyzing their variability. These approaches are known for their flexibility and accuracy, particularly in capturing epistemic uncertainty [24]. As mentioned earlier, for high-dimensional real-valued parameters \(\theta\) such as those in neural networks, the integral in Equation 1 becomes intractable and is typically approximated using Monte Carlo samples:

\begin{equation} P(y \mid x, \mathcal{D}) \approx \frac{1}{N} \sum_{k=1}^{N} P(y \mid x, \theta^{(k)}); \quad \theta^{(k)} \sim p(\theta \mid \mathcal{D})\qquad(2) \label{eq:ensemble_approximation} \end{equation}

based on \(N\) different sets of parameters \(\theta^{(k)}\). Different sampling-based methods differ in how they compute these parameters. Some approaches rely on explicit posterior approximations, such as Bayesian Neural Networks [5], while others use implicit sampling techniques like MC-Dropout [2] or Deep Ensembles [1]. Sampling-based approaches require multiple predictions, and some also require training several models. Deep Ensembles are a strong practical baseline, including under dataset shift [27], but the tradeoff between cost and uncertainty quality depends on the task and evaluation setup.

Deep Ensembles and MC-Dropout

Deep Ensembles train independently initialized networks and combine their predictions [1]. MC-Dropout instead keeps dropout active at inference and evaluates the same input under different random masks [2]. Both expose variation across predictions, but they differ in the source of that variation and in training and memory cost.

Snapshot and Fast Geometric Ensembles

Snapshot Ensembles collect checkpoints from a single training trajectory [3]. Fast Geometric Ensembling uses low-loss connections between solutions to obtain additional ensemble members [4]. These approaches reduce the cost of obtaining members compared with training each independently, while still requiring multiple predictions at inference.

Masksembles

Masksembles [7] enhance MC-Dropout by carefully designing structured masks that selectively drop specific features. This approach increases diversity among predictions, resulting in improved uncertainty quantification.

BatchEnsembles

BatchEnsembles [6] offer an efficient ensembling approach by representing multiple models within a single network using learnable rank-one perturbations of the weights. This design minimizes memory overhead while preserving diversity in predictions. By sharing parameters across models, BatchEnsembles strike a balance between scalability and uncertainty estimation quality.

Subnetwork Inference

Subnetwork Inference [13] is a scalable Bayesian framework that performs posterior inference over a small subset of neural network weights, keeping the rest as point estimates. By performing inference on a selected subnetwork with expressive approximations like full-covariance Gaussians, it aims to balance computational efficiency with accurate uncertainty estimation.

Packed-Ensembles

Packed-Ensembles [8] propose an efficient ensembling approach that uses grouped convolutions to create multiple independent subnetworks within a single shared architecture. The model is trained similarly to a standard single model but produces multiple predictions, mimicking ensembling while introducing only minor computational overhead.

MIMO

The Multiple Input Multiple Output framework (MIMO) [9], along with similar approaches like MixMO [28], enables efficient ensembling by training independent subnetworks within a single model. During training, \(N\) inputs are processed simultaneously, allowing the model to make \(N\) independent predictions in a single forward pass. At inference, the input is replicated \(N\) times to generate \(N\) predictions, which are ensembled to estimate uncertainty.

Iterative Uncertainty

The Iterative Uncertainty framework [10] leverages the iterative refinement process of neural networks to estimate uncertainty without modifying the original architecture. By analyzing the variance in outputs across consecutive iterations, it uses the rate of convergence as a proxy for certainty, where faster convergence indicates higher confidence. A related concept, Prediction Depth [12], measures the layer depth at which a model’s prediction stabilizes, with harder examples requiring deeper layers for convergence.

ABNN

Adaptable Bayesian Neural Networks (ABNN) [11], along with similar approaches like BatchNorm Uncertainty [14], transform pre-trained DNNs into Bayesian Neural Networks by fine-tuning the model’s normalization layers and introducing noise during inference to approximate sampling.

Bayes by Backprop

Bayes by Backprop [5] is a variational inference method that approximates the posterior distribution of model parameters by minimizing a variational objective. Instead of learning point estimates, it learns a distribution over weights, typically using a Gaussian variational posterior.

Sampling-Free Methods

Sampling-free methods avoid repeated stochastic sampling or independently trained ensemble members. Some produce an uncertainty estimate in one forward pass; others, such as ZigZag, use a small fixed number of deterministic evaluations. As mentioned earlier, sampling-based approaches are often too costly for real-world applications like robotics and autonomous driving, necessitating faster alternatives. These methods achieve efficiency through architectural modifications, deterministic approximations, or specialized loss functions. Several popular approaches are discussed in the following subsections.

Distance-Aware Uncertainty

Recent approaches such as Deterministic Uncertainty Quantification (DUQ) [17] and SNGP [16] focus on single–forward-pass uncertainty estimation by introducing distance-aware output layers, such as radial basis functions (RBFs) in DUQ and Gaussian processes (GPs) in SNGP. DUQ employs a Jacobian penalty to encourage smoothness in the feature space, while SNGP uses spectral normalization to ensure sensitivity and stability in the latent space. Building on these ideas, Deep Deterministic Uncertainty (DDU) [18] further extends the framework by directly modeling the feature density in the latent space.

Conformal Prediction

Conformal prediction [26] constructs prediction sets or intervals with marginal coverage guarantees under assumptions such as exchangeability of the calibration and test data. These guarantees concern average coverage, rather than certainty for every individual input. It operates by calibrating model outputs on a held-out dataset to produce prediction intervals or sets, making it agnostic to the underlying model architecture. This flexibility allows conformal prediction to be applied across a wide range of tasks, offering a principled and reliable approach to uncertainty quantification in real-world applications.

Second-Order Probabilistic Modeling

Second-order probabilistic modeling approaches—such as prior networks [22], posterior networks [23], and evidential learning [21]—capture uncertainty by modeling distributions over predictive probabilities. These methods leverage a factorization of the predictive distribution, as shown in Equation 3, to decompose uncertainty into aleatoric, distributional, and epistemic components.

\[p(y \mid x, \mathcal{D}) = \iint \underbrace{P(y \mid \pi)}_{\text{Aleatoric}} \, \underbrace{p(\pi \mid x, \theta)}_{\text{Distributional}} \, \underbrace{p(\theta \mid \mathcal{D})}_{\text{Epistemic}} \, d\pi \, d\theta \approx \\ \approx \int \underbrace{P(y \mid \pi)}_{\text{Aleatoric}} \, \underbrace{p(\pi \mid x, \hat{\theta})}_{p(\theta \mid \mathcal{D}) \approx \delta(\theta - \hat{\theta})} \, d\pi.\qquad(3)\]

Prior networks directly model the distributional uncertainty \(p(\pi \mid x, \hat{\theta})\) by parameterizing a Dirichlet distribution, enabling the network to represent confidence levels tied to predictive uncertainty. Posterior networks extend this by incorporating learned feature-level distributions, combining evidence from the data to produce refined predictions. Evidential learning follows a similar framework, interpreting input data as evidence to parameterize distributions over class probabilities, often using a Dirichlet or related family.

Additional Information Approaches

Recent methods leverage auxiliary information to probe model confidence, based on the idea that confident models remain unaffected by additional inputs, while uncertain ones show prediction shifts. ZigZag [19] measures uncertainty by comparing predictions with and without auxiliary information (ground truth during training and model predictions during inference), revealing sensitivity to added data. Similarly, Experts Don’t Cheat [20] leverages paired responses, allowing models to “cheat” by using one response to predict the other, thereby exposing uncertain predictions.

Variance Propagation and Orthonormal Certificates

Variance propagation [24] approximates how uncertainty passes through network layers instead of estimating it with repeated stochastic predictions. Orthonormal certificates [25] take a different route: they learn functions that are close to zero on training examples and use deviations on new inputs as a signal of unfamiliarity. Both reduce the need for repeated sampling, while relying on their own approximations or assumptions.

Conclusion

Uncertainty estimation is a choice about what information to extract and how much computation to spend. Bayesian inference provides a common starting point, but practical methods approximate it in different ways—or use alternative signals such as feature density, convergence, or sensitivity to auxiliary information. A single uncertainty score does not mean the same thing across all these approaches.

Deep Ensembles and MC-Dropout offer useful multiple-prediction baselines; structured ensembles and partial Bayesian inference reduce some of their costs. Deterministic methods trade repeated sampling for architectural or modeling assumptions, while conformal prediction provides calibrated sets or intervals under explicit data assumptions. The most suitable approach therefore depends on the task, latency and memory budget, and the kinds of errors or distribution shifts that matter.

In practice, compare predictive performance and uncertainty quality together, and test on both familiar data and realistic shifts [27]. A confident prediction is useful only when confidence remains informative about error. The goal is to support decisions about when to trust a prediction, when to collect more data, and when to defer to another source of information.

References

[1] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. NeurIPS, 2017.

[2] Y. Gal and Z. Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. ICML, 2016.

[3] G. Huang et al. Snapshot Ensembles: Train 1, get M for free. ICLR, 2017.

[4] T. Garipov et al. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. NeurIPS, 2018.

[5] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight Uncertainty in Neural Networks. ICML, 2015.

[6] Y. Wen, D. Tran, and J. Ba. BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning. ICLR, 2020.

[7] N. Durasov, T. Bagautdinov, P. Baqué, and P. Fua. Masksembles for Uncertainty Estimation. CVPR, 2021.

[8] O. Laurent et al. Packed-Ensembles for Efficient Uncertainty Estimation. ICLR, 2023.

[9] M. Havasi et al. Training independent subnetworks for robust prediction. ICLR, 2021.

[10] N. Durasov, D. Öner, J. Donier, H. Le, and P. Fua. Enabling Uncertainty Estimation in Iterative Neural Networks. ICML, 2024.

[11] G. Franchi et al. Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models. CVPR, 2024.

[12] R. J. N. Baldock, H. Maennel, and B. Neyshabur. Deep Learning Through the Lens of Example Difficulty. NeurIPS, 2021.

[13] E. Daxberger, E. Nalisnick, J. U. Allingham, J. Antorán, and J. M. Hernández-Lobato. Bayesian Deep Learning via Subnetwork Inference. ICML, 2021.

[14] A. Atanov et al. Uncertainty Estimation via Stochastic Batch Normalization. 2018.

[15] A. Kendall and Y. Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?. NeurIPS, 2017.

[16] J. Z. Liu et al. Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness. NeurIPS, 2020.

[17] J. van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal. Uncertainty Estimation Using a Single Deep Deterministic Neural Network. ICML, 2020.

[18] J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. S. Torr, and Y. Gal. Deep Deterministic Uncertainty: A Simple Baseline. 2021 preprint.

[19] N. Durasov, N. Dorndorf, H. Le, and P. Fua. ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference. TMLR, 2024.

[20] D. D. Johnson, D. Tarlow, D. Duvenaud, and C. J. Maddison. Experts Don’t Cheat: Learning What You Don’t Know By Predicting Pairs. ICML, 2024.

[21] M. Sensoy, L. Kaplan, and M. Kandemir. Evidential Deep Learning to Quantify Classification Uncertainty. NeurIPS, 2018.

[22] A. Malinin and M. Gales. Predictive Uncertainty Estimation via Prior Networks. NeurIPS, 2018.

[23] B. Charpentier, D. Zügner, and S. Günnemann. Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts. NeurIPS, 2020.

[24] J. Postels et al. Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagation. ICCV, 2019.

[25] N. Tagasovska and D. Lopez-Paz. Single-Model Uncertainties for Deep Learning. NeurIPS, 2019.

[26] G. Shafer and V. Vovk. A Tutorial on Conformal Prediction. JMLR, 2008.

[27] Y. Ovadia et al. Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. NeurIPS, 2019.

[28] A. Ramé, R. Sun, and M. Cord. MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks. ICCV, 2021.