Gaussian Likelihoods in Bayesian Neural Networks

Abstract

Bayesian neural networks (BNNs) offer a promising probabilistic take on neural networks, allowing uncertainty quantification in both model predictions and parameters. Being a relatively new and evolving field of research, many aspects of Bayesian neural networks still need to be better understood. In this thesis, we explore the Gaussian likelihood function commonly used when modeling regression problems with Bayesian neural networks. Using variational inference, we train several Bayesian neural networks on synthetic datasets and investigate the Gaussian variance parameter (sigma). We explore how it impacts the training process and shapes the resulting posterior distribution. We also explore an alternate approach where a prior distribution is placed on the variance parameter, and its value is inferred from the data. While the data presented in this thesis is too limited to draw any definitive conclusions, we provide some interesting insights. We demonstrate that extreme values for sigma can lead to tendencies of overfitting or underfitting BNNs. Additionally, inferring the variance parameter from the data can yield results on par with an "optimal" fixed parameterization of the likelihood function. We also showcase that misspecified Bayesian neural networks can produce overconfident uncertainty estimates and that inferring the variance parameter can help compensate for this limitation.