I got to see Sean Talts and Michael Betancourt giving very good (and crowded , ) workshops at PyData NYC this past week, and it got me to hacking on a PyMC3 version of the algorithm from their recent paper (also with Dan Simpson, Aki Vehtari, and Andrew Gelman).
The paper provides an algorithm, simulation based calibration (SBC), for checking whether an algorithm that produces samples from a posterior (like MCMC, ADVI, or INLA) might work for a given model. This calibration is independent of the observations for a model:
- we sample parameters
θfrom the prior, then use those samples to generate draws from the prior predictive distribution.
- For each draw
yfrom the prior predictive distribution, we calculate, say, 100 draws from our posterior, using
yas our observation.
- If you sort
θinto the posterior samples, its position (“rank statistic”) should be uniformly distributed.
Read the paper. It is very nice.
Using the library
You can pip install and use the library right now!
pip install git+https://github.com/ColCarroll/simulation_based_calibration
See the github project for sample usage.