Referring Image Segmentation by Generative Adversarial Learning

Qiu, Shuang; Zhao, Yao; Jiao, Jianbo; Wei, Yunchao; Wei, Shikui

Abstract

Referring expression is a kind of language expression being used for referring to particular objects. In this paper, we focus on the problem of image segmentation from natural language referring expressions. Existing works tackle this problem by augmenting the convolutional semantic segmentation networks with an LSTM sentence encoder, which is optimized by a pixel-wise classification loss. We argue that the distribution similarity between the inference and ground truth plays an important role in referring image segmentation. Therefore we introduce a complementary loss considering the consistency between the two distributions. To this end, we propose to train the referring image segmentation model in a generative adversarial fashion, which well addresses the distribution similarity problem. In particular, the proposed adversarial semantic guidance network (ASGN) includes the following advantages: a) more detailed visual information is incorporated by the detail enhancement; b) semantic information counteracts the word embedding impact; c) the proposed adversarial learning approach relieves the distribution inconsistencies. Experimental results on four standard datasets show significant improvements over all the compared baseline models, demonstrating the effectiveness of our method.

Citation

Qiu, S., Zhao, Y., Jiao, J., Wei, Y., & Wei, S. (2020). Referring Image Segmentation by Generative Adversarial Learning. IEEE Transactions on Multimedia, 22(5), 1333-1344. https://doi.org/10.1109/tmm.2019.2942480

• Journal Article Type: Article
• Online Publication Date: Sep 20, 2019
• Publication Date: 2020-05
• Deposit Date: Jun 7, 2023
• Journal: IEEE Transactions on Multimedia
• Print ISSN: 1520-9210
• Electronic ISSN: 1941-0077
• Publisher: Institute of Electrical and Electronics Engineers (IEEE)
• Peer Reviewed: Peer Reviewed
• Volume: 22
• Issue: 5
• Pages: 1333-1344
• DOI: https://doi.org/10.1109/tmm.2019.2942480
• Keywords: Electrical and Electronic Engineering; Computer Science Applications; Media Technology; Signal Processing