Multimodal Distributions in Variational Autoencoders

In this thesis, I studied possibility of usage of multimodal distributions in variational au- toencoders. Recent breakthroughs in deep learning have resulted in new neural network architectures that allow to solve a wide range of tasks. Among them are autoencoders which are designed to learn latent representation of data, usually of lower dimension. Further im- provements of the model include usage of variational inference when assumptions are made regarding distribution of latent variables, and the resulting model is called variational au- toencoder (VAE). However, special tricks have to be applied in order to efficiently learn the model, and not the whole range of distributions can be used in VAEs due to this. At the same time more complex assumptions on distribution in latent space allow to obtain richer representation and are able to better describe the data.

This work presents a new VAE with loguniform prior and spike and slab posterior distri- bution which is a mixture of Dirac Delta function and a Gaussian probability function. Such choice of distributions promotes learning of sparse latent representation and allows to obtain a more complex latent space structure comparing with VAE with just Gaussian posterior. Since ordinary reparametrization trick is not applicable in this case, a special procedure for backpropagation through mixture weights was derived and implemented for efficient learning of the model.

In a series of experiments the model suggested above demonstrated applicability and allowed to obtain sparse representation of data from different datasets. In addition to that, obtained sparsity was expressed both in terms of elimination of the whole dimensions for all objects at the same time and in terms of individual sparsity where different dimensions were eliminated for different objects.

Recommended citation: E. Iakovleva. "Multimodal Distributions in Variational Autoencoders." MSc Thesis at Skoltech, 2017.
Download Paper