收稿：2021-04-18，
纸质出版：2021-12-07
移动端阅览
通过拍照进行三维建模的摄影测量技术是文物数字化的重要手段，然而，在摄影测量文物三维重建过程中，文物所在的场景背景也同时参与计算和建模，需要设法删除，以获得文物单体模型。针对上述问题，本文提出一种小型文物三维模型主体对象识别方法，能够自动删除文物摄影测量三维模型的背景数据。该方法分别利用深度学习网络Mask R-CNN和改进为自动获得初值的One Cut进行文物图像前景内容识别，再融合两种方法的识别结果进行文物图像主体对象分割，最后以二维图像分割结果为依据，逐三角形识别三维模型主体对象，并运用多视角约束法优化识别精度。实验表明，本文方法能够获得小型文物单体化三维模型，无需人工干预且精度较高。
Photogrammetry technology helps us reconstruct three dimensional models of cultural relics just by taking photos. However
the background where the cultural relics are located also participates in modeling simultaneously
which wastes storage space and computing resources. Meanwhile
the independence and aesthetics of the three dimensional models are destroyed. Additionally
pure models of cultural relics are obtained by manually deleting the background in three dimensional scenes
which is time consuming and cannot satisfy the practical needs of the flourishing development of digital cultural heritage.
This research aims to obtain the three dimensional pure cultural relic models by deleting the redundant background of the photogrammetric model on the basis of object recognition without manual interaction.
This paper proposed a method to delete the background of the three dimensional photogrammetric model of cultural relics by objects recognition. First
we recognized the foreground of the cultural relic image by using the deep learning network Mask R-CNN and One Cut
respectively. Second
we extracted the masks of cultural relics by combining the results of Mask R-CNN and One Cut. Last
we applied the masks of cultural relics to delete the background of three dimensional cultural relic models on the basis of the mapping relationships between images and three dimensional models. Moreover
we used the multi-view constraints to optimize the three dimensional recognition accuracy. Additionally
we improved the One Cut method by automatically setting the initial value. In the processing of three dimensional projecting to two dimensional
regarding the cases where triangles overlap
we applied the depth information to distinguish the triangles of foreground and background in three dimensional models.
To evaluate proposed method
two cultural relics were selected for the experiments
including Buddha statues in the Beilin Museum in Shaanxi and Mayan masks in the Mexican Museum. We took photos of them and obtained three dimensional models via GET3D (get3d. cn). Our method performs effectively for the Buddha model and the Mayan masks model. Apparently
most of the background of the models is eliminated
and the main bodies of the models are completely preserved. Compared with the artificially labeled ground truth
it can be found that 1) our method preserved three dimensional models complete with a satisfactory recall of 99.23% and 99.20% for the Buddha model and the Mayan masks model
respectively; 2) the algorithm erased the triangles of background with a simplification rate of 85.34% and 86.44% for the Buddha model and the Mayan masks model
respectively; 3) with the advantage of the multi-view constraints
the recognition accuracy of the three dimensional model is higher than two dimensional image.
The method proposed in this paper can automatically delete the background of the three dimensional photogrammetric model without manual intervention and preserve the integrity of the object well. The experimental results demonstrate the proposed method is feasible and effective. However
when applied to large three dimensional models
our method is limited to efficiency
given that we distinguished the overlapped triangles successively. Moreover
our pipeline provides a reference for recognizing three dimensional objects in various three dimensional scenes.
Aldoma A , Tombari F , Di Stefano L and Vincze M . 2012 . A global hypotheses verification method for 3D object recognition //Lazebnik A, Fitzgibbon S, Perona P, Sato Y and Schmidt C, eds. Computer Vision - ECCV 2012 . Florence, Italy : Springer : 511 - 524 [ DOI: 10.1007/978-3-642-33712-3_37 http://dx.doi.org/10.1007/978-3-642-33712-3_37 ]
Borji A . 2015 . What is a salient object? A dataset and a baseline model for salient object detection. IEEE Transactions on Image Processing , 24 ( 2 ): 742 - 756 [ DOI: 10.1109/tip.2014.2383320 http://dx.doi.org/10.1109/tip.2014.2383320 ]
Boykov Y , Veksler O and Zabih R . 2001a . Fast approximate energy minimization via graph cuts . IEEE Transactions on Pattern Analysis and Machine Intelligence , 23 ( 11 ): 1222 - 1239 [ DOI: 10.1109/34.969114 http://dx.doi.org/10.1109/34.969114 ]
Boykov Y Y and Jolly M P . 2001b . Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images // Proceedings Eighth IEEE International Conference on Computer Vision . Vancouver, BC, Canada : IEEE : 105 - 112 [ DOI: 10.1109/ICCV.2001.937505 http://dx.doi.org/10.1109/ICCV.2001.937505 ]
Carvalho L E and von Wangenheim A . 2019 . 3D object recognition and classification: a systematic literature review . Pattern Analysis and Applications , 22 ( 4 ): 1243 - 1292 DOI: 10.1007/s10044-019-00804-4 http://dx.doi.org/10.1007/s10044-019-00804-4 ]
Chen H T . 2010 . Preattentive co-saliency detection // 2010 IEEE International Conference on Image Processing . Hong Kong, China : IEEE : 1117 - 1120 [ DOI: 10.1109/ICIP.2010.5650014 http://dx.doi.org/10.1109/ICIP.2010.5650014 ]
Ch’ng E , Cai S D , Zhang T E and Leow F T . 2019 . Crowdsourcing 3D cultural heritage: best practice for mass photogrammetry . Journal of Cultural Heritage Management and Sustainable Development , 9 ( 1 ): 24 - 42 [ DOI: 10.1108/JCHMSD-03-2018-0018 http://dx.doi.org/10.1108/JCHMSD-03-2018-0018 ]
Cong R M , Lei J J , Fu H Z , Cheng M M , Lin W S and Huang Q M . 2019 . Review of visual saliency detection with comprehensive information . IEEE Transactions on Circuits and Systems for Video Technology , 29 ( 10 ): 2941 - 2959 [ DOI: 10.1109/tcsvt.2018.2870832 http://dx.doi.org/10.1109/tcsvt.2018.2870832 ]
Fu K R , Gu I Y H , Yun Y X , Gong C and Yang J . 2014 . Graph construction for salient object detection in videos // 2014 22nd International Conference on Pattern Recognition . Stockholm, Sweden : IEEE : 2371 - 2376 [ DOI: 10.1109/ICPR.2014.411 http://dx.doi.org/10.1109/ICPR.2014.411 ]
Gong Y P , Zhang F , Jia X Y , Huang X F , Li D R and Mao Z . 2021 .“ Deep Neural Networks for Quantitative Damage Evaluation ofBuilding Losses Using Aerial Oblique Images: Case Study on theGreat Wall (China) .” Remote Sensing 13 ( 7 ): 19 [ DOI: 10.3390/rs13071321 http://dx.doi.org/10.3390/rs13071321 ]
Guo Y L , Bennamoun M , Sohel F , Lu M and Wan J W . 2014 . 3D object recognition in cluttered scenes with local surface features: a survey . IEEE Transactions on Pattern Analysis and Machine Intelligence , 36 ( 11 ): 2270 - 2287 [ DOI: 10.1109/TPAMI.2014.2316828 http://dx.doi.org/10.1109/TPAMI.2014.2316828 ]
Guo Y L , Wang H Y , Hu Q Y , Liu H , Liu L and Bennamoun M . 2020 . Deep learning for 3D point clouds: a survey . IEEE Transactions on Pattern Analysis and Machine Intelligence [ DOI: 10.1109/tpami.2020.3005434 http://dx.doi.org/10.1109/tpami.2020.3005434 ]
Han J W , Zhang D W , Cheng G , Liu N and Xu D . 2018 . Advanced deep-learning techniques for salient and category-specific object detection: a survey . IEEE Signal Processing Magazine , 35 ( 1 ): 84 - 100 [ DOI: 10.1109/msp.2017.2749125 http://dx.doi.org/10.1109/msp.2017.2749125 ]
He K M , Gkioxari G , Dollár P and Girshick R . 2017 . Mask R-CNN // 2017 IEEE International Conference on Computer Vision . Venice, Italy : IEEE : 2980 - 2988 [ DOI: 10.1109/ICCV.2017.322 http://dx.doi.org/10.1109/ICCV.2017.322 ]
Jakab M , Benesova W and Racev M . 2015 . 3D object recognition based on local descriptors // Proceedings Volume 9406, Intelligent Robots and Computer Vision XXXII: Algorithms and Techniques . San Francisco, California, United States : SPIE : 94060 L [ DOI: 10.1117/12.2083104 http://dx.doi.org/10.1117/12.2083104 ]
Li D P , Wang H Y , Liu N , Wang X M and Xu J . 2020 . 3D object recognition and pose estimation from point cloud using stably observed point pair feature . IEEE Access , 8 : 44335 - 44345 [ DOI: 10.1109/ACCESS.2020.2978255 http://dx.doi.org/10.1109/ACCESS.2020.2978255 ]
Li H L , Meng F M and Ngan K N . 2013 . Co-salient object detection from multiple images . IEEE Transactions on Multimedia , 15 ( 8 ): 1896 - 1909 [ DOI: 10.1109/tmm.2013.2271476 http://dx.doi.org/10.1109/tmm.2013.2271476 ]
Lin T Y , Maire M , Belongie S , Hays J , Perona P , Ramanan D , Dollár P and Zitnick C L . 2014 . Microsoft COCO: common objects in context //Fleet D, Pajdla T, Schiele B and Tuytelaars T, eds. Computer Vision - ECCV 2014 . Zurich, Switzerland : Springer : 740 - 555 [ DOI: 10.1007/978-3-319-10602-1_48 http://dx.doi.org/10.1007/978-3-319-10602-1_48 ]
Loaiza C , Daniel A , María M M and Gabriel M B . 2020 . “ Virtual Museums . Captured Reality and 3 d Modeling.” Journal of Cultural Heritage 45 : 234 - 9 [ DOI: 10.1016/j.culher.2020.04.013 http://dx.doi.org/10.1016/j.culher.2020.04.013 ]
Luo Y , Yuan J S and Lu J W . 2016 . Finding spatio-temporal salient paths for video objects discovery . Journal of Visual Communication and Image Representation , 38 : 45 - 54 [ DOI: 10.1016/j.jvcir.2016.02.001 http://dx.doi.org/10.1016/j.jvcir.2016.02.001 ]
Peng H W , Li B , Xiong W H , Hu W M and Ji R R . 2014 . RGBD salient object detection: a benchmark and algorithms //Fleet D, Pajdla T, Schiele B and Tuytelaars T, eds. Computer Vision - ECCV 2014 . Zurich, Switzerland : Springer : 92 - 109 [ DOI: 10.1007/978-3-319-10578-9_7 http://dx.doi.org/10.1007/978-3-319-10578-9_7 ]
Qu L Q , He S F , Zhang J W , Tian J D , Tang Y D and Yang Q X . 2017 . RGBD salient object detection via deep fusion . IEEE Transactions on Image Processing , 26 ( 5 ): 2274 - 2285 [ DOI: 10.1109/TIP.2017.2682981 http://dx.doi.org/10.1109/TIP.2017.2682981 ]
Rother C , Kolmogorov V and Blake A . 2004 . “GrabCut”: interactive foreground extraction using iterated graph cuts . ACM Transactions on Graphics , 23 ( 3 ): 309 - 314 [ DOI: 10.1145/1015706.1015720 http://dx.doi.org/10.1145/1015706.1015720 ]
Shen Z Q , Ma X and Li Y B . 2018 . A hybrid 3D descriptor with global structural frames and local signatures of histograms . IEEE Access , 6 : 39261 - 39272 [ DOI: 10.1109/ACCESS.2018.2856866 http://dx.doi.org/10.1109/ACCESS.2018.2856866 ]
Singh R D , Mittal A and Bhatia R K . 2019 . 3D convolutional neural network for object recognition: a review . Multimedia Tools and Applications , 78 ( 12 ): 15951 - 15995 [ DOI: 10.1007/s11042-018-6912-6 http://dx.doi.org/10.1007/s11042-018-6912-6 ]
Sun X , Yang B S and Li Q Q . 2011 . “ Structural Segmentation Method for 3d Building Models Based on Voxel Analysis .” Acta Geodeticaet Cartographica Sinica 40 ( 5 ): 582 - 6
孙轩 , 杨必胜 , 李清泉 . 2011 . 基于体元分析的三维建筑物模型结构化分割方法 . 测绘学报 , 40 ( 05 ): 582 - 586
Tang M , Gorelick L , Veksler O and Boykov Y . 2013 . GrabCut in one cut // 2013 IEEE International Conference on Computer Vision . Sydney, NSW, Australia : IEEE : 1769 - 1776 [ DOI: 10.1109/ICCV.2013.222 http://dx.doi.org/10.1109/ICCV.2013.222 ]
Ullah I , Jian M W , Hussain S , Guo J , Yu H , Wang X and Yin Y L . 2020 . A brief survey of visual saliency detection . Multimedia Tools and Applications , 79 ( 45 ): 34605 - 34645 [ DOI: 10.1007/s11042-020-08849-y http://dx.doi.org/10.1007/s11042-020-08849-y ]
Vetrivel A. , M. Gerke , N. Kerle , and G. Vosselman . 2015 . Segmentation of Uav-Based Images Incorporating 3d Point Cloud Information. Paper presented at the Joint ISPRS Conference on Photogrammetric Image Analysis (PIA) and High Resolution Earth Imaging for Geospatial Information (HRIGI), Technische UnivMunchen, Munich, GERMANY, Mar 25 - 27 [ DOI: 10.5194/isprsarchives-XL-3-W2-261-2015 http://dx.doi.org/10.5194/isprsarchives-XL-3-W2-261-2015 ]
Wen W W , Wen G J , Hui B W and Qiu S H . 2018 . 3D object recognition based on improved point cloud descriptors // Proceedings Volume 10806 , Tenth International Conference on Digital Image Processing (ICDIP 2018 . Shanghai, China: SPIE: 108060 O [ DOI: 10.1117/12.2503095 http://dx.doi.org/10.1117/12.2503095 ]
Yan Y M , Gao F J , Deng S P , and Su N . 2017 . “ A Hierarchical Building Segmentation in Digital Surface Models for 3d Reconstruction .” Sensors 17 ( 2 ) [DOI: 10.3390/s17020222]
Yang C , Zhang F. , Gao Y L , Mao Z , Li L , and Huang X F . 2021 . “ Moving Car Recognition and Removal for 3d Urban Modelling Using Oblique Images .” Remote Sensing 13 ( 17 ): 19 [ DOI: 10.3390/rs13173458 http://dx.doi.org/10.3390/rs13173458 ]
Zhao Z Q , Zheng P , Xu S T and Wu X D . 2019 . Object detection with deep learning: a review . IEEE Transactions on Neural Networks and Learning Systems , 30 ( 11 ): 3212 - 3232 [ DOI: 10.1109/tnnls.2018.2876865 http://dx.doi.org/10.1109/tnnls.2018.2876865 ]
相关作者
相关机构
京公网安备11010802024621
