Semantic inpainting on segmentation map via multi-expansion loss

Jianfeng He, Xuchao Zhang, Shuo Lei

Abstract

Semantic Inpainting on Segmentation Map (SISM) aims to manipulate segmentation maps by semantics. Recent works show SISM provides semantic-aware auxiliary information for better style or structure manipulations. Providing structural assistance, segmentation maps have been broadly used as an intermediate interface to achieve better image manipulation. Mainstream solutions of image manipulation use Generative Adversarial Net (GAN) globally, locally or jointly. It is also applicable to SISM. However, the discriminator of global GAN is easier fooled, because the majority of its input is the same as the ground-truth, which is hard to fully mitigate the inconsistency between inpainted areas and the context. The inconsistency is more difficult for local GAN to address, due to the lack of context in its input. To mitigate the inconsistency, we propose a novel Multi-Expansion (MEx) loss. It is implemented by the adversarial loss on MEx areas. Each MEx area has the inpainted area as dominance and keeps knowledge of the scene context, so the consistency of the SISM results can be boosted. We propose an approximation of MEx loss, i.e., A-MEx loss, to further enhance the stability and usability. Besides performing well on SISM tasks, MEx loss also performs impressively on natural image inpainting. Extensive experiments on the two tasks demonstrate the advantages of our model over existing methods on four challenging datasets, such as a 2.59% increase in Hamm on SISM in Cityscape and a decrease of 5.00% FID on natural image inpainting in CMP Facade. The code of our work is available at: https://github.com/he159ok/AMEx-MEx-Loss.