Modern machine learning methods
for trustworthy science
Tom Charnock
Institut d'Astrophysique de Paris
Why neural networks don't work
(and how to use them)
Tom Charnock
Institut d'Astrophysique de Paris
Why neural networks don't work
Tom Charnock
Institut d'Astrophysique de Paris
Apologies about the term bias¶
when something is intrinsically unknowable it is biased
if there is some offset, which could in principle be corrected, it is biased
I (almost always) mean the top one¶
A crazy likelihood surface of how likely we are to get targets from data¶
What are we actually interested in?¶
How likely are the true targets given some data?
How likely are the targets to be generated by a particular network?
What is the probability of obtaining a particular network with particular parameter values?
Where does this information about the weights and hyperparameters come from?¶
Training and validation data¶
Training data and targets: $\small\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{train}\equiv\{{\bf d}^\textrm{train}_i, {\bf t}^\textrm{train}_i|i\in[1,n_\textrm{train}]\}$
Validation data and targets: $\small\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{val}\equiv\small\{{\bf d}^\textrm{val}_i, {\bf t}^\textrm{val}_i|i\in[1,n_\textrm{val}]\}$
Posterior distribution of weights and hyperparameters¶
$$\begin{align*} &\small\mathcal{P}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{train},\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{val})\propto\\ &\small\phantom{hello}\mathcal{L}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{train},\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{val})p(\boldsymbol{\omega},\boldsymbol{\alpha}) \end{align*}$$The failing of traditional training¶
$\mathcal{M} : {\bf d} \to {\bf t}$ approximator
$$\mathbb{NN}(\boldsymbol{\omega}, \boldsymbol{\alpha}) : {\bf d}\to\boldsymbol{\tau}$$
Cost function and likelihood $$\boldsymbol{\Lambda}({\bf t}, \boldsymbol{\tau}) = -\ln\mathcal{L}({\bf t}|{\bf d},\boldsymbol{\omega}^*,\boldsymbol{\alpha}^*)$$
smooth and convex
$\mathcal{L}({\bf t}|{\bf d},\boldsymbol{\omega},\boldsymbol{\alpha})$
complex and non-convex in $\boldsymbol{\omega}$ and $\boldsymbol{\alpha}$
Optimising (or training) a network¶
What are the maximum likelihood estimates of the weights?¶
Local maximum likelihood estimates¶
The main problem...¶
We degenerate the posterior¶
$$\begin{align*}
\mathcal{P}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{train})\propto&~\mathcal{L}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{train})p(\boldsymbol{\omega},\boldsymbol{\alpha})\\
\to&~\delta(\boldsymbol{\omega}-\boldsymbol{\omega}^\textrm{MLE},\boldsymbol{\alpha}-\boldsymbol{\alpha}^*)
\end{align*}$$
All predictions are (probably incorrect) estimates
$$\mathcal{P}({\bf t}|{\bf d})=\delta(\boldsymbol{\tau})$$
There is no way to interpret how close $\boldsymbol{\tau}$ is to ${\bf t}$...¶
Because the likelihood is non-interpretably complex¶
Are there better methods?¶
Variational inference¶
Still depends on fixed weights in the complex likelihood surface
and choice of variational distribution
$$\begin{align*}
\mathcal{P}({\bf t}|{\bf d})&=\int d\boldsymbol{\omega}d\boldsymbol{\nu}d\boldsymbol{\alpha}~\mathcal{L}({\bf t}|{\bf d},\boldsymbol{\omega},\boldsymbol{\alpha})\mathcal{Q}(\boldsymbol{\omega}|\boldsymbol{\nu}, \boldsymbol{\alpha}, \boldsymbol{\{}{\bf d}, {\bf t}\boldsymbol{\}}^\textrm{train})\\
&\phantom{hellohello}\times\delta(\boldsymbol{\nu}-\boldsymbol{\nu}^\textrm{MLE}, \boldsymbol{\alpha}-\boldsymbol{\alpha}^*)\\
&=\int d\boldsymbol{\omega}~\mathcal{L}({\bf t}|{\bf d},\boldsymbol{\omega}, \boldsymbol{\alpha}^*)\mathcal{Q}(\boldsymbol{\omega}|\boldsymbol{\nu}^\textrm{MLE}, \boldsymbol{\alpha}^*, \boldsymbol{\{}{\bf d}, {\bf t}\boldsymbol{\}}^\textrm{train}).
\end{align*}$$
Bayesian neural networks¶
Sample the likelihood of the training data¶
Still dependent on the training data!¶
Classical network : $\tiny \mathcal{P}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{train}) \to \delta(\boldsymbol{\omega}-\boldsymbol{\omega}^\textrm{MLE},\boldsymbol{\alpha}-\boldsymbol{\alpha}^*)$
Variational inference : $\tiny \mathcal{P}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{train}) = \mathcal{Q}(\boldsymbol{\omega}|\boldsymbol{\nu}^\textrm{MLE}, \boldsymbol{\alpha}^*, \boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{train})$
Bayesian networks : $\tiny \mathcal{P}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},{\bf t}\boldsymbol{\}}^\textrm{train}) =\prod_i^{n_\textrm{train}}\mathcal{L}({\bf t}^\textrm{train}_i|{\bf d}^\textrm{train}_i,\boldsymbol{\omega},\boldsymbol{\alpha})p(\boldsymbol{\omega},\boldsymbol{\alpha})$
Problems with physical models...¶
How can we use a neural network then?¶
Build it into the physical model¶
Method 1 :
Infer the data, physics and the neural network
Method 2 :
Understand the likelihood (using neural physical engines)
Method 3 :
Likelihood-free inference
Compare distance between observed summaries and simulation summaries and select results within $\epsilon$
