Auto-Encoding Variational Bayes · CiteArk

Auto-Encoding Variational Bayes

Public

Authors:Diederik P. Kingma, Max Welling

arXiv 2013 Machine Learning (stat.ML)Machine Learning (cs.LG)

Research summary

Based on the paper

This paper uses reparameterization to make sampling of continuous latent variables differentiable, so the inference and generative networks can optimize one variational objective. It compares AEVB with wake-sleep on MNIST and Frey Face, and compares marginal-likelihood estimates on low-dimensional MNIST.

What this research means

Research interpretation

The shared differentiable training objective provides the core training method behind variational auto-encoders.

AEVB’s lower-bound advantage survives reconstruction, but likelihood rankings depend on conditions

This MNIST reconstruction shows two evidence patterns: final ELBO has a consistent direction across dimensions, whereas the HMC marginal-likelihood ranking changes with training scale. The dimensionality probe establishes only finite estimates, not an accuracy boundary.

Metric and scale shape the reconstruction

In the reconstructed MNIST curves, AEVB has higher final train and test ELBO than wake-sleep at all five latent dimensions. Yet the final test marginal-likelihood leader changes from wake-sleep with 1,000 training examples to AEVB with 50,000, while MCEM is lowest in both; finite probes at 3D, 5D, and 10D cannot validate the paper’s low-dimensional reliability boundary without ground truth.

View values and explanation

AEVB leads wake-sleep in final train and test ELBO at all five recorded dimensions; final test marginal likelihood is led by wake-sleep with 1,000 examples and by AEVB with 50,000, while the dimensionality probe cannot validate an accuracy boundary.

Final test ELBO gap at 20D (AEVB minus wake-sleep): 4.337466
Final test ELBO gap at 200D (AEVB minus wake-sleep): 4.443145
Final test mean log marginal likelihood for AEVB, 50,000 examples: -148.5871
Final test mean log marginal likelihood for wake-sleep, 1,000 examples: -159.2733
Paper-reported strict dimension boundary (not validated by probe): 5