Stain normalization with GANs

Hi ,

Stain normalization is only one technique for combating variations across labs and scanners that can degrade model performance, but it is probably the most widely used. (See my previous newsletter on domain shift for a discussion of other approaches.)

It is also the topic that clients most commonly ask me about.

Traditionally, methods like color matching [1] and stain separation [2] were used. These methods rely on selection of a reference slide; however, Ren et al. showed that using an ensemble with different reference slides is one possible solution [3].

The larger problem is that these techniques do not consider spatial features, which can lead to tissue structure not being preserved.

Generative Adversarial Networks (GANs) are the state-of-the-art in stain normalization today. Given an image from domain A, a generator converts it into domain B. A discriminator network tries to distinguish real domain B images from fake ones, helping the generator to improve.

Aligned Image Solution

If paired and aligned images from domains A and B are available, this setup performs well. However, it typically requires scanning each slide on two different scanners -- or potentially even restaining and rescanning each slide.

But there is a simpler solution to obtaining paired images: convert a color image to grayscale (domain A) and pair it with the original color image (domain B) [4]. The two are perfectly aligned and a Conditional GAN can be trained to reconstruct the color image.

One major advantage of this approach is that a restaining model trained for one particular domain may work for a variety of different labs and scanners as there is less variation in the input grayscale images than in color.

Preserving Tissue Structure

An alternative approach when paired images are not available is a CycleGAN. In this setup there are two generators: one to convert from domain A to B and another to go from domain B to A. The goal of these two models is to be able to reconstruct an original image: A -> B -> A or B -> A -> B. CycleGANs also makes use of discriminators to predict real versus generated images for each domain.

CycleGAN performance can be improved further by incorporating semantic knowledge to better preserve tissue structure. Mahapatra et al. provided the CycleGAN with features from a pretrained segmentation model and found that it helped to preserve cellular structure [5]. The pretrained model didn’t even need to be from histology -- however, performance improved an additional 1% when it was.

a) Domain A, b) Domain B, c) Mahapatra et al. d) and e) alternative methods with arrows identifying structure inconsistency [5]

Learning the Residual

Finally, I want to tell you about a way to improve both a Conditional GAN when paired images are available and a CycleGAN for when they are not: residual learning.

If you’re familiar with ResNet or other architectures that make use of skip connections, it’s a similar idea here. You provide a shortcut connection across network layers, including one directly from the input to the output, and sum the features. This means that the network is not learning how to reconstruct domain B from domain A, but instead is learning the difference between the domains -- likely a simpler task.

Residual connection to learn difference between source and target domain [7]

Nishar et al. applied this to a Conditional GAN when they only had 20 paired images and found training to be much faster than alternative models [6]. The paired images were not even of the exact same tissue but of tissue with similar content.

For CycleGANs, de Bel et al. demonstrated improved segmentation results after applying their model with residual learning [7].

Recommendations

Whether you're working with a Conditional GAN or CycleGAN, residual learning can enable faster training with fewer training images required. And for extra help in preserving tissue structure, try incorporating semantic knowledge with a pretrained segmentation model.

I hope that these techniques provide some insights into how you can upgrade your stain normalization algorithms.

But remember, they are most effective when used in combination with color augmentation and domain adaptation for improved generalization performance.