Auto-Encoding Variational Bayes

Figure 5 presents random samples from learned MNIST generative models at each of four latent-space dimensionalities: 2, 5, 10, and 20.

The authors report that the relative performance of the compared algorithms was not very sensitive to the chosen hidden-layer widths.

The paper states that a learned two-dimensional AEVB encoder can project MNIST and Frey Face observations into a low-dimensional latent space; Figure 4 itself shows grids of decoded model outputs p_theta(x|z) at inverse-Gaussian-CDF-transformed latent coordinates, not plotted encoded datapoint projections.

In Figure 3 with 1,000 MNIST training examples, the AEVB training marginal-likelihood curve rises faster than MCEM and the two approach similar displayed training endpoints; the displayed MCEM test curve finishes above AEVB, while wake-sleep has lower train and test curves than both near the end.

Increasing the number of latent variables, including apparently superfluous variables, did not produce overfitting in the Figure 2 experiments; the paper attributes this to regularization by the variational lower bound.

In Figure 3 with 50,000 MNIST training examples, AEVB has the highest displayed train and test marginal-likelihood curves, wake-sleep is lower, and MCEM improves much later and remains lower over the plotted training range.

The Figure 2 computations took approximately 20-40 minutes per million training samples on an Intel Xeon CPU operating at an effective 40 GFLOPS.

Across every lower-bound comparison shown in Figure 2, AEVB converged considerably faster than wake-sleep and reached a better variational-lower-bound solution on both training and test curves.

For the Figure 2 lower-bound curves, the paper reports that estimator variance was below 1 and omits it from the plot.

The Figure 3 marginal-likelihood estimates were computed on the first 1,000 datapoints of both the training and test sets, using 50 posterior samples per datapoint from HMC with 4 leapfrog steps.

The MCEM baseline used 10 HMC leapfrog steps with automatically tuned step size targeting a 90% acceptance rate, followed by 5 parameter-weight update steps; all compared algorithms used Adagrad step sizes with an annealing schedule.

The marginal-likelihood estimator is reported to give good estimates only when the sampled latent space is very low-dimensional and enough samples are used; the experiment therefore used 3 latent variables, while estimates at higher dimensionality became unreliable.

In the reported experiments, one latent sample per datapoint was found sufficient when the minibatch contained about 100 datapoints.