Method of Semi-supervised Video Object Segmentation Based on Encoding Memory Network

YIN Liang; ZHANG Zhao; ZHANG Baopeng

doi:10.15892/j.cnki.djzdxb.2024.03.002

Journal of Projectiles, Rockets, Missiles and Guidance >

2024 , Vol. 44 >Issue 3: 11 - 21

DOI: https://doi.org/10.15892/j.cnki.djzdxb.2024.03.002

Method of Semi-supervised Video Object Segmentation Based on Encoding Memory Network

YIN Liang ¹ ,
ZHANG Zhao ² ,
ZHANG Baopeng ²

Expand

¹ CHN Energy Xinshuo Railway Co., Ltd. Maintenance Branch office, Beijing 010300, China
² School of Computer and Information Technology, Beijing Jiaotong University, Beijing 100044, China

Received date: 2024-02-20

Online published: 2024-12-28

Fold

Abstract

Video object segmentation is a key task in computer vision and is of great significance to fields such as autonomous driving and video coding. For the video object segmentation, the proposed method utilizes an efficient encoding memory network (EMNet) to achieve semi-supervised video object segmentation. The method includes an adaptive reference frame selection module, a dual path matching module, a feature processing module and a feature aggregation module. The adaptive reference frame selection module takes into account mask confidence and similarity, and selects a reference frame that contains rich information. The dual-path matching module realizes bidirectional and dual-scale matching between query frames and reference frames to improve the accuracy of target feature matching. The feature processing module includes a semantic enhancement module and a feature refinement module, which enhance the semantic and detailed information of the target through low-pass and high-pass filtering. Finally, the feature aggregation module fuses and utilizes each feature. An evaluation is carried out on the DAVIS2017dataset and the result shows that the proposed method is effective.

Key words： video object segmentation; encoding memory network; attention mechanism; semantic segmentation; deep learning

Cite this article

YIN Liang , ZHANG Zhao , ZHANG Baopeng . Method of Semi-supervised Video Object Segmentation Based on Encoding Memory Network[J]. Journal of Projectiles, Rockets, Missiles and Guidance, 2024 , 44(3) : 11 -21 . DOI: 10.15892/j.cnki.djzdxb.2024.03.002

[an error occurred while processing this directive]

References

Publishing order | Descend order by publishing year | Descend order by cited within

[1]	SIAM M, MAHGOUB H, ZAHRAN M, et al. Video object segmentation using speaker identification[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2018: 7386-7395.

[2]	LEE Y J, KIM J, GRAUMAN K. Key-segments for video object segmentation[C]// IEEE. Proceedings of the IEEE International Conference on Computer Vision. New York: IEEE, 2011: 1995-2002.

[3]	LI Y, YE Z, REHG J M. Delving into egocentric actions[C]// IEEE. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015: 287-295.

[4]	HE K, GKIOXARI G, DOLLÁR P, et al. Mask r-cnn[C]// IEEE. Proceedings of the IEEE International Conference on Computer Vision. New York: IEEE, 2017: 2961-2969.

[5]	PERAZZI F, KHOREVA A, BENENSON R, et al. Learning video object segmentation from static images[C]// IEEE. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 2663-2672.

[6]	TSAI Y H, YANG M H, BLACK M J. Video segmentation via object flow[C]// IEEE. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2016: 3899-3908.

[7]	OH S W, LEE J Y, XU N, et al. Video object segmentation using space-time memory networks[C]// IEEE. Proceedings of the IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2019: 9226-9235.

[8]	JAIN M, JÉGOU H, BOUTHEMY P. Better exploiting motion for better action recognition[C]// IEEE. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2013: 2555-2562.

[9]	MCFARLANE N J, SCHOFIELD C P. Segmentation and tracking of piglets in images[J]. Machine Vision and Applications, 1995, 8(3): 187-193.

[10]	GRUNDMANN M, KWATRA V, HAN M, et al. Efficient hierarchical graph-based video segmentation[C]// IEEE. Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2010: 2141-2148.

[11]	OCHS P, BROX T. Higher order motion models and spectral clustering[C]// IEEE. Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2012: 614-621.

[12]	ZHANG Z, HUANG K, TAN T. Comparison of similarity measures for trajectory clustering in outdoor surveillance scenes[C]// IEEE. Proceedings of the 18th International Conference on Pattern Recognition. New York: IEEE, 2006: 1135-1138.

[13]	HU W, HU M, ZHOU X, et al. Principal axis-based correspondence between multiple cameras for people tracking[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(4): 663-671. PMID

[14]	JAMPANI V, GADDE2 R, GEHLER P V. Video propagation networks[C]// IEEE. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 451-461.

[15]	YANG L, WANG Y, XIONG X, et al. Efficient video object segmentation via network modulation[C]// IEEE. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2018: 6499-6507.

[16]	VOIGTLAENDER P, CHAI Y, SCHROFF F, et al. Feelvos: fast end-to-end embedding learning for video object segmentation[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2019: 9481-9490.

[17]	CAELLES S, MANINIS K K, PONT-TUSET J, et al. One-shot video object segmentation[C]// IEEE. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 221-230.

[18]	LUITEN J, VOIGTLAENDER P, LEIBE B. Premvos: proposal-generation, refinement and merging for video object segmentation[C]// ACCV. Proceedings of the 2018 Asian Conference on Computer Vision. Berlin:Springer International Publishing, 2018: 565-580.

[19]	OH S W, LEE J Y, SUNKAVALLI K, et al. Fast video object segmentation by reference-guided mask propagation[C]// IEEE. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2018: 7376-7385.

[20]	WANG Z, XU J, LIU L, et al. Ranet: ranking attention network for fast video object segmentation[C]// IEEE. Proceedings of the IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2019: 3978-3987.

[21]	WANG H, JIANG X, REN H, et al. Swiftnet: real-time video object segmentation[C]// IEEE. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2021: 1296-1305.

[22]	SEONG H, HYUN J, KIM E. Kernelized memory network for video object segmentation[C]// ECCV. Proceedings of the 2020 Computer Vision-ECCV: 16th European Conference.Berlin:Springer International Publishing, 2020: 629-645.

[23]	IPPOLITI L, VALENTINI P, GAMERMAN D. Space-time modelling of coupled spatiotemporal environmental variables[J]. Journal of the Royal Statistical Society Series C: Applied Statistics, 2012, 61(2): 175-200.

[24]	LIN Z, YANG T, LI M, et al. Swem: Towards real-time video object segmentation with sequential weighted expectation-maximization[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2022: 1362-1372.

[25]	HU Y T, HUANG J B, SCHWING A G. Videomatch: Matching based video object segmentation[C]// ECCV. Proceedings of the 2018 European conference on computer vision (ECCV). Berlin:Springer International Publishing, 2018: 54-70.

[26]	HU L, ZHANG P, ZHANG B, et al. Learning position and target consistency for memory-based video object segmentation[C]// IEEE. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2021: 4144-4154.

[27]	XU X, WANG J, LI X, et al. Reliable propagation-correction modulation for video object segmentation[C]// AIAA. Proceedings of the AIAA Conference on Artificial Intelligence. Reston: AIAA, 2022: 2946-2954.

[28]	ZHANG Y, ROBINSON A, MAGNUSSON M, et al. Flow-guided semi-supervised video object segmentation[C]// CVPR. Proceedings of the 2023 IEEE Computer Vision and Pattern Recognition Conference. New York: IEEE, 2023: 10492.

[29]	SEONG H, HYUN J, KIM E. Kernelized memory network for video object segmentation[C]// ECCV. Proceedings of the 2020 Computer Vision-ECCV 16th European Conference. Berlin:Springer International Publishing, 2020: 629-645.

Options

Outlines