Simulation Based Calibration in PyMC3

I got to see Sean Talts and Michael Betancourt giving very good (and crowded [1], [2]) workshops at PyData NYC this past week, and it got me to hacking on a PyMC3 version of the algorithm from their recent paper (also with Dan Simpson, Aki Vehtari, and Andrew Gelman).

The paper provides an algorithm, simulation based calibration (SBC), for checking whether an algorithm that produces samples from a posterior (like MCMC, ADVI, or INLA) might work for a given model. This calibration is independent of the observations for a model:

• we sample parameters θ from the prior, then use those samples to generate draws from the prior predictive distribution.
• For each draw y from the prior predictive distribution, we calculate, say, 100 draws from our posterior, using y as our observation.
• If you sort θ into the posterior samples, its position (“rank statistic”) should be uniformly distributed.

Read the paper. It is very nice.

Using the library

You can pip install and use the library right now!

pip install git+https://github.com/ColCarroll/simulation_based_calibration

See the github project for sample usage.