Epistemic Uncertainty Quantification in Deep Learning by the Delta Method
MetadataVis full innførsel
This thesis explores the Delta method and its application to deep learning image classification. The Delta method is a classical procedure for quantifying uncertainty in statistical models, but its direct application to deep neural networks is prevented by the large number of parameters P. We recognize the Delta method as a measure of epistemic as opposed to aleatoric uncertainty and break it into two components: the eigenvalue spectrum of the inverse Fisher information (i.e. inverse Hessian) of the cost function and the per-example sensitivities (i.e. gradients) of the model function. We mainly focus on the computational aspects, and show how to efficiently compute low and full-rank approximations of the inverse Fisher information matrix, which in turn reduces the computational complexity of the naïve Delta method from O(P²) space and O(P³) time, to O(P) space and time. We provide bounds for the approximation error by a novel error propagating technique, and validate the developed methodology with a released TensorFlow implementation. By a comparison with the classical Bootstrap, we show that there is a strong linear relationship between the quantified predictive epistemic uncertainty levels obtained from the two methods when applied on a few well known architectures using the MNIST and CIFAR-10 datasets.
Består avPaper 1: Geir K. Nilsen, Antonella Z. Munthe-Kaas, Hans J. Skaug and Morten Brun, Efficient Computation of Hessian Matrices in TensorFlow, arXiv preprint: 1905.05559, 2019, revised 2021. The article is available in the thesis file. The article is also available at: https://doi.org/10.48550/arXiv.1905.05559
Paper 2: Geir K. Nilsen, Antonella Z. Munthe-Kaas, Hans J. Skaug and Morten Brun, Epis- temic Uncertainty Quantification in Deep Learning Classification by the Delta Method, Neural Networks, 2022, 145: 164-176. The article is available at: https://hdl.handle.net/11250/2835021
Paper 3: Geir K. Nilsen, Antonella Z. Munthe-Kaas, Hans J. Skaug, Morten Brun, A Com- parison of the Delta Method and the Bootstrap in Deep Learning Classification, arXiv preprint: 2107.01606, 2021. The article is available in the thesis file. The article is also available at: https://doi.org/10.48550/arXiv.2107.01606