Neural networks
How they work and their future in science

Tom Charnock

Institut d'Astrophysique de Paris

What is a neural network?¶

Neural networks are arbitrary fittable non-linear functions¶

$$\mathscr{f} : {\bf d}\to\boldsymbol{\tau}$$

There are many different architectures¶

Convolutional neural networks

Recurrent neural networks

For any given dataset there is generally an obvious choice of architecture

How to fit a neural network¶

Approximate a function, $\mathscr{g} : {\bf d}\to\boldsymbol{\theta}$,
with a neural network, $\mathscr{f} : {\bf d}\to\boldsymbol{\tau}$

Probabilistic interpretation of a neural network¶

How likely is any network going to provide the expected function output?

This surface is defined by our choice in distance measure which is called the cost or loss function

Obtaining informative outputs¶

$$\mathcal{P}(\boldsymbol{\theta}|{\bf d})= \int d\boldsymbol{\omega}d\boldsymbol{\alpha}~\mathcal{L}(\boldsymbol{\theta}|{\bf d},\boldsymbol{\omega},\boldsymbol{\alpha})\mathcal{P}(\boldsymbol{\omega},\boldsymbol{\alpha})$$

Maximum likelihood estimate of weights¶

$$\boldsymbol{\omega}^\textrm{MLE}=\underset{\boldsymbol{\omega}}{\textrm{argmax}}\left[\mathcal{L}(\boldsymbol{\{}\boldsymbol{\theta}\boldsymbol{\}}^\textrm{train}|\boldsymbol{\{}{\bf d}\boldsymbol{\}}^\textrm{train}, \boldsymbol{\omega}, \boldsymbol{\alpha}^*)\right]$$

Prediction using neural networks¶

Why do we use neural networks?¶

They're quick and (relatively) cheap¶

Like how FFT are more efficient than directly evaluating Fourier Transforms¶

Neural networks can fit extremely complex functions to run blazingly quickly¶

Super-resolution and super-approximation of dark matter simulations¶

Emulating high resolution N-body simulations

Inpainting high quality observable quantities

Neural networks are driven by data¶

i.e. they can be fit with data where there is no description via a physical model¶

Generating images of faces

Detecting tumours for aided cancer discovery

Physical models from data models¶

Data models are the description that include all effects that generate the data¶

Initial conditions

Physical model

Instrumental effects

Other stochasticity

Neural networks are great at learning the data model¶

We can build neural networks to deal with parts of the data model we don't understand¶

Extracting physical models from the data¶

Correlations between interesting observables and the processes that create them¶

Can we build neural networks that are informative about these processes?¶

Neural physical engines¶

Building symmetries into neural networks¶

Decompose neural networks into well structured, orthogonal, algorithms¶

Find what part of the data is informative about a given result¶

Use visualising attention to perturb the network to find out what is most sensitive

Because of the orthogonal nature of the neural physical engine we can relate physical structures in the data to outcomes from the trained network

Statistical understanding of neural networks¶

There is no sense in trust in how correct any output of the network is to be correct

Choosing a neural network with weights at their maximum likelihood estimates degenerates the posterior¶

Local maximum likelihood estimates¶

Different results are obtained with each network, none with a sense of trust

No interpretability and no trust = no science¶

Bayesian neural networks and variational inference¶

Posterior predictive distribution of outputs¶

Everything is dependent on the training data!¶

Classical network : $\tiny \mathcal{P}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},\boldsymbol{\theta}\boldsymbol{\}}^\textrm{train}) \to \delta(\boldsymbol{\omega}-\boldsymbol{\omega}^\textrm{MLE},\boldsymbol{\alpha}-\boldsymbol{\alpha}^*)$

Variational inference : $\tiny \mathcal{P}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},\boldsymbol{\theta}\boldsymbol{\}}^\textrm{train}) = \mathcal{Q}(\boldsymbol{\omega}|\boldsymbol{\nu}^\textrm{MLE}, \boldsymbol{\alpha}^*, \boldsymbol{\{}{\bf d},\boldsymbol{\theta}\boldsymbol{\}}^\textrm{train})$

Bayesian networks : $\tiny \mathcal{P}(\boldsymbol{\omega},\boldsymbol{\alpha}|\boldsymbol{\{}{\bf d},\boldsymbol{\theta}\boldsymbol{\}}^\textrm{train}) =\prod_i^{n_\textrm{train}}\mathcal{L}(\boldsymbol{\theta}^\textrm{train}_i|{\bf d}^\textrm{train}_i,\boldsymbol{\omega},\boldsymbol{\alpha})p(\boldsymbol{\omega},\boldsymbol{\alpha})$

If the training data, network optimisation, or (pretty much) anything else is not perfect, the inference will be biased¶

Likelihood-free inference¶

We have a model with which we can simulate the data¶

Physical model
Generative models based on neural networks

We cannot (easily) describe the data¶

The likelihood of obtaining data is unknown/complex
There isn't even a well defined model

Use the same neural network to summarise observations and simulations¶

Generate simulations drawn from prior and summarise¶

Make a kernel density estimate (or use ABC) on the simulations¶

A slice through at the summarised observation is the approximate posterior

Posterior distribution of galaxy counts and fluxes in fields

Summary¶

Neural networks are extremely powerful for science¶

They speed up computations¶

They can also allow us to build model-independent models¶

When used carefully, and in the right setting they give us exact statistically interpretable results¶

And in the future they will actually inform us on how to build physical models¶

Extracting information using neural networks¶

Information maximising neural networks¶

Which function $\mathscr{f}: \mathbb{R}^{\bf d}\to \mathbb{R}^\boldsymbol{\theta}$ maximises the Fisher information of the summaries ${\bf x}$ from that function?

$$\begin{align} {\bf F}_{\alpha\beta} &= \left.\left\langle\frac{\partial^2\ln\mathcal{L}({\bf d}|\boldsymbol{\theta})}{\partial\theta_\alpha\partial\theta_\beta}\right\rangle\right|_{\boldsymbol{\theta}=\boldsymbol{\theta}^*}\nonumber\\ &\simeq \left.\left\langle\frac{\partial^2\ln\mathcal{L}({\bf x}|\boldsymbol{\theta}, {\bf d})}{\partial\theta_\alpha\partial\theta_\beta}\right\rangle\right|_{\boldsymbol{\theta}=\boldsymbol{\theta}^*}\nonumber \end{align}$$

And also Gaussianises the data $$\mathcal{L}({\bf x}|\boldsymbol{\theta},{\bf d})\approx\mathcal{N}(\boldsymbol{\mu}(\boldsymbol{\theta}), \boldsymbol{\Sigma}(\boldsymbol{\theta}))$$

We maximise¶

$$\ln|{\bf F}_{\alpha\beta}| = \ln\left|\frac{\partial\boldsymbol{\mu}_\mathscr{f}}{\partial\theta_\alpha}^T{\bf C}^{-1}_\mathscr{f}\frac{\partial\boldsymbol{\mu}_\mathscr{f}}{\partial\theta_\beta}\right|.$$

Calculate the covariance¶

Calculate the derivative of the mean of the summaries with respect to the parameters¶

Calculate the Fisher information¶

$${\bf F}_{\alpha\beta}=\frac{\partial\mu_\mathscr{f}}{\partial\theta_\alpha}^T{\bf C}^{-1}_\mathscr{f}\frac{\partial\mu_\mathscr{f}}{\partial\theta_\beta}$$

This Gaussian form forces the summaries to be Gaussianised

And use LFI¶

These are the optimal summaries to extract information from the data

We have lots of examples of how powerful this tool is throughout scientific domains¶

Neural networksHow they work and their future in science