Hard computation is a well-known problem in various ML algorithms today, especially when generative AI is applied to text, images, and other unstructured data.
One of the main ways to reduce this problem is to compress the input data into a low-dimensional representation while preserving the original content. There are various methods that achieve this goal, including autoencoders, which we will discuss in this article.
For simplicity, in this article, we will focus on image-based autoencoders, but remember, they can be used for other types of data as well.
An idea
Autoencoders are a type of neural network used for unsupervised learning. Their structure consists of three main parts:
- Encoder. The first part of the neural network that takes the input data and gradually reduces its size through its layers, finally reaching the threshold.
- A bottle. A very small-dimensional network layer that contains a discrete representation of the input data.
- Decoder. The part of the network connected to the output bottleneck that gradually expands the size of the data. As a result, its last layer, returns data of the same size as it was originally passed to the encoder.
In graphics, the encoder and decoder are often presented as convolutional neural networks.
For autoencoders, our main goal during training is to make the network convert the input data into a highly compressed representation at the bottleneck without losing much information. During decoding, we can pass the data to the encoder, remove the resulting embedding from the bottleneck, and use it for our purposes.
Let's understand how training works on autoencoders.
Training
The great thing about autoencoders is that they don't need labeled data! Let's see how they work.
As mentioned before, the input image is transmitted to the network, where it is compressed to a smaller size and then reconstructed to the original size. The question we should ask ourselves is what we want the decoder to output.
As you might have guessed, the decoder can simply try to reconstruct the original image from the compressed image in the bottle. Why should it?
The idea behind this is simple:
- If the compressed bottleneck template correctly captures the main features of the encoder input, it should be easy for the decoder to use that information to reconstruct the original image.
- If the bottleneck fails to capture key features, the decoder will not be able to reliably reconstruct the original image. Therefore, the model will be penalized for poorly compressed representation.
In this way, by asking the decoder to reconstruct the original image, we implicitly force the encoder to produce a rich but compressed latent representation, which helps the encoder to perform its task effectively.
The space where the input data is displayed in the bottleneck is called hidden place.
Loss of reconstruction
Given the original image and the reconstructed image from the decoder, what is the easiest way to compare the quality produced? The obvious answer is to compare two images pixel-wise using Loss of MSEthat, in the context of autoencoders, is called i loss of reconstruction.
The calculated loss value is then used to perform backpropagation to update the model weights.
The size of the latent space
The size of the hidden space is an important hyperparameter that directly affects the performance of the decoder.
On the other hand, the size of the latent space should be sufficient to encode effectively the main input features. On the other hand, it should not be too big to maintain a high pressure level.
One well-known example is Stable Diffusion. It uses an autoencoder to convert the input image, 512 x 512 x 3containing 786,432 values, be a 64 x 64 x 4 picture with 16,384 values, resulting in a compression ratio of 48x.
Some autoencoder applications
One technique for training autoencoders is to learn how to remove noise from images. The idea is simple: since autoencoders can reconstruct original images, we can add a little noise to the input images and ask them to reconstruct the original images.
The good thing about this method is that for training, it is enough to have only real images, where you will add sound.
Another cool application of autoencoders is image painting, which involves passing images with masked patches to the model so that it can decode and fill in the missing parts of the image.
Similarly, autoencoders can be trained to remove certain objects from images. This is especially useful for removing watermarks.
The problem of greenness
In fact, despite its simplicity, MSE loss is not perfect for autoencoders. One common problem with using it is the tendency for the encoder to produce images with blurry pixels.
For example, we might think of a picture of size 512 x 512 with two direct and non-overlapping regions. Then we take a horizontal row of that image whose pixels look like this:
[… 0 0 255 255 255 …]
The model has no knowledge of the image structure; it only tries to minimize MSE loss. Even if, in that picture, the model makes such a prediction [… 0 0 0 255 255 …]which is still very good because the region is shifted by only one pixel, the MSE loss in this case can be higher than in the case below, the model may choose:
[… 0 0 127 255 255 …]
In the latter case, despite the low MSE, the middle pixel represents a fuzzy, poorly visible edge.
This problem is solved by a more advanced autoencoder variant with modified loss functions.
The conclusion
As we can see, autoencoders are a simple but powerful concept. By training the decoder to reconstruct the original image from the compressed data, we gradually adjust the encoder to generate informative features that can be extracted for downstream tasks.
In addition to data compression, we have seen that autoencoders have other applications, such as removing noise from images, performing image painting, and extracting objects from images.
All photos unless otherwise stated by the author
