视觉SLAM机器人中光束法平差优化芯片研究综述

莫霄睿; 张惟宜; 年成; 郭与时; 牛丽婷; 张柏雯; 张春

doi:10.20193/j.ices2097-4191.2024.0038

PDF(16052 KB)

集成电路与嵌入式系统 ›› 2024, Vol. 24 ›› Issue (11) : 29-40. DOI: 10.20193/j.ices2097-4191.2024.0038

智能机器人高能效专用芯片研究专栏

视觉SLAM机器人中光束法平差优化芯片研究综述

莫霄睿 ¹ ,
张惟宜 ¹ ,
年成 ¹ ,
郭与时 ¹ ,
牛丽婷 ¹ ,
张柏雯 ² ,
张春 ¹^,^*

作者信息 +

A survey of bundle adjustment optimization chips in visual SLAM robots

MO Xiaorui ¹ ,
ZHANG Weiyi ¹ ,
NIAN Cheng ¹ ,
GUO Yushi ¹ ,
NIU Liting ¹ ,
ZHANG Baiwen ² ,
ZHANG Chun ¹^,^*

Author information +

文章历史 +

摘要

在视觉同时定位与地图构建(Visual Simultaneous Localization and Mapping, V-SLAM)系统中,光束法平差(Bundle Adjustment, BA)是优化相机参数和三维点位置的重要环节。然而,由于BA计算复杂度高,实时性要求高,传统的计算平台难以满足高效计算的需求。近年来,专用硬件加速器的引入为BA优化提供了新的解决方案。本文综述了BA优化专用芯片的研究现状及发展趋势,主要涵盖了BA算法的应用场景、定义与基本原理;BA在现场可编程门阵列(Field-Programmable Gate Arrays, FPGA)、专用集成电路(Application-Specific Integrated Circuits, ASIC)和图形处理单元(Graphics Processing Units, GPU)上的加速方法,以及这些加速器的发展趋势。此外,本文还探讨了BA加速器在技术实现中面临的挑战,并展望了其未来的发展方向。

Abstract

In Visual Simultaneous Localization and Mapping (V-SLAM) systems, Bundle Adjustment (BA) plays a crucial role in optimizing camera parameters and the positions of 3D points. However, due to the high computational complexity and real-time requirements of BA, traditional computing platforms struggle to meet efficiency demands. Recently, the introduction of dedicated hardware accelerators has provided new solutions for BA optimization. This paper reviews the current status of research and development trends in BA optimization-specific chips. It covers the application scenarios, definitions, and basic principles of BA algorithms; the acceleration of BA on Field-Programmable Gate Arrays (FPGA), Application-Specific Integrated Circuits (ASIC), and Graphics Processing Units (GPU), as well as the development trends of these accelerators. Furthermore, this paper discusses the technical challenges in implementing BA accelerators and anticipates future development directions. By summarizing current research advancements, this review aims to provide guidance and insights for future studies on BA optimization-specific chips.

导出引用

莫霄睿, 张惟宜, 年成, 等. 视觉SLAM机器人中光束法平差优化芯片研究综述[J]. 集成电路与嵌入式系统. 2024, 24(11): 29-40 https://doi.org/10.20193/j.ices2097-4191.2024.0038

MO Xiaorui, ZHANG Weiyi, NIAN Cheng, et al. A survey of bundle adjustment optimization chips in visual SLAM robots[J]. Integrated Circuits and Embedded Systems. 2024, 24(11): 29-40 https://doi.org/10.20193/j.ices2097-4191.2024.0038

中图分类号： TN492 (专用集成电路)

参考文献

列表( 原文顺序 | 文献年度倒序 | 文中引用次数倒序 ) 可视化分析

[1]	YANG T, LI P, ZHANG H, et al. Monocular vision SLAM-based UAV autonomous landing in emergencies and unknown environments[J]. Electronics, 2018, 7(5):73. 本文引用 [1]

[2]	ZHAO J, LIANG B, CHEN Q. The key technology toward the self-driving car[J]. International Journal of Intelligent Unmanned Systems, 2018, 6(1):2-20.

[3]	LIU S. Engineering autonomous vehicles and robots:the dragonfly modular-based approach[M].John Wiley & Sons, 2020.

[4]	HAN J, CHO Y, KIM J. Coastal SLAM with marine radar for USV operation in GPS-restricted situations[J]. IEEE Journal of Oceanic Engineering, 2019, 44(2):300-309. 本文引用 [1]

[5]	KISS-ILLÉS D, BARRADO C, SALAMÍ E. GPS-SLAM:An augmentation of the ORB-SLAM algorithm[J]. Sensors, 2019, 19(22):4973. 本文引用 [1]

[6]	SULEIMAN A, ZHANG Z, CARLONE L, et al. Navion:A 2-mw fully integrated real-time visual-inertial odometry accelerator for autonomous navigation of nano drones[J]. IEEE Journal of Solid-State Circuits, 2019, 54(4):1106-1119. 本文引用 [4]

[7]	ZENG F, WANG C, GE S S. A survey on visual navigation for artificial agents with deep reinforcement learning[J]. IEEE Access, 2020(8):135426-135442. 本文引用 [1]

[8]	CHAN S H, WU P T, FU L C. Robust 2D indoor localization through laser SLAM and visual SLAM fusion[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2018:1263-1268. 本文引用 [1]

[9]	ERVAN O, SENCAN O, TEMELTAS H. Transversal positioned (TRAP) 2D laser range finder configuration[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2016:1-6. 本文引用 [1]

[10]	LATEGAHN H, GEIGER A, KITT B. Visual SLAM for autonomous ground vehicles[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC. IEEE,2011:1732-1737. 本文引用 [1]

[11]	YOON J S, KIM J H, KIM H E, et al. A unified graphics and vision processor with a 0.89/spl mu/W/fps pose estimation engine for augmented reality[J]. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 2012, 21(2):206-216. 本文引用 [3]

[12]	高翔. 视觉SLAM十四讲:从理论到实践[M]. 北京: 电子工业出版社, 2017. GAO X. Visual SLAM Lecture 14:from theory to practice[M]. Beijing: Electronic Industry Press, 2017 (in Chinese). 本文引用 [3]

[13]	MUR-ARTAL R, MONTIEL J M M, TARDOS J D. ORB-SLAM:a versatile and accurate monocular SLAM system[J]. IEEE transactions on robotics, 2015, 31(5):1147-1163. 本文引用 [2]

[14]	TRIGGS B, MCLAUCHLAN P F, HARTLEY R I, et al. Bundle adjustment-a modern synthesis[C]// Vision Algorithms:Theory and Practice: International Workshop on Vision Algorithms Corfu,Greece, September 21-22,1999 Proceedings.Springer Berlin Heidelberg,2000: 298-372. 本文引用 [2]

[15]	MUR-ARTAL R, TARDÓS J D. Orb-slam2:An open-source slam system for monocular, stereo, and rgb-d cameras[J]. IEEE transactions on robotics, 2017, 33(5):1255-1262. 本文引用 [3]

[16]	ERIKSSON A, BASTIAN J, CHIN T J, et al. A consensus-based framework for distributed bundle adjustment[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2016:1754-1762. 本文引用 [1]

[17]	ZHANG R, ZHU S, FANG T, et al. Distributed very large scale bundle adjustment by global camera consensus[C]// Proceedings of the IEEE International Conference on Computer Vision, 2017:29-38. 本文引用 [1]

[18]	BRESSON G, ALSAYED Z, YU L, et al. Simultaneous localization and mapping:A survey of current trends in autonomous driving[J]. IEEE Transactions on Intelligent Vehicles, 2017, 2(3):194-220. 本文引用 [1]

[19]	SONG W, LIU S, TONG X, et al. Experimental Comparison and Analysis of Block Bundle Adjustment Models for Chinese ZY-3 Optical Satellite Imagery[C]// IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium.IEEE,2019: 9055-9058. 本文引用 [1]

[20]	MALINVERNI E S, PIERDICCA R, BOZZI C A, et al. Evaluating a SLAM-based mobile mapping system: a methodological comparison for 3D heritage scene real-time reconstruction[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2018:265-270. 本文引用 [1]

[21]	VACA-CASTANO G, ZAMIR A R, SHAH M. City scale geo-spatial trajectory estimation of a moving camera[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2012:1186-1193. 本文引用 [1]

[22]	MAIMONE M, CHENG Y, MATTHIES L. Two years of visual odometry on the mars exploration rovers[J]. Journal of Field Robotics, 2007, 24(3):169-186. 本文引用 [1]

[23]	KLINGNER B, MARTIN D, ROSEBOROUGH J. Street view motion-from-structure-from-motion[C]// Proceedings of the IEEE International Conference on Computer Vision,2013:953-960. 本文引用 [1]

[24]	ZHANG Z. Determining the epipolar geometry and its uncertainty:A review[J]. International journal of computer vision, 1998,27:161-195. 本文引用 [1]

[25]	BESL P J, MCKAY N D. Method for registration of 3-D shapes[C]// Sensor fusion IV: control paradigms and data structures.Spie, 1992,1611:586-606. 本文引用 [1]

[26]	PAN S, WANG X. A survey on perspective-n-point problem[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE 40th Chinese Control Conference (CCC). IEEE,2021: 2396-2401. 本文引用 [1]

[27]	CHEN G, LI L, JIN W, et al. High-dynamic range,night vision,image-fusion algorithm based on a decomposition convolution neural network[J]. IEEE Access, 2019(7):169762-169772. 本文引用 [1]

[28]	GAI S, JUNG E J, YI B J. Multi-group localization problem of service robots based on hybrid external localization algorithm with application to shopping mall environment[J]. Intelligent Service Robotics, 2016(9):257-275. 本文引用 [1]

[29]	HIJIKATA S, TERABAYASHI K, UMEDA K. A simple indoor self-localization system using infrared LEDs[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2009:1-7.

[30]

MÜLLER

, RASSWEILER

M C

, KLEIN

, et al. Mobile augmented reality for computer-assisted percutaneous nephrolithotomy[J]. International journal of computer assisted radiology and surgery, 2013, 8: 663-675.

https://doi.org/10.1007/s11548-013-0828-4

https://www.ncbi.nlm.nih.gov/pubmed/23526436

摘要

Percutaneous nephrolithotomy (PCNL) plays an integral role in treatment of renal stones. Creating percutaneous renal access is the most important and challenging step in the procedure. To facilitate this step, we evaluated our novel mobile augmented reality (AR) system for its feasibility of use for PCNL.A tablet computer, such as an iPad[Formula: see text], is positioned above the patient with its camera pointing toward the field of intervention. The images of the tablet camera are registered with the CT image by means of fiducial markers. Structures of interest can be superimposed semi-transparently on the video images. We present a systematic evaluation by means of a phantom study. An urological trainee and two experts conducted 53 punctures on kidney phantoms.The trainee performed best with the proposed AR system in terms of puncturing time (mean: 99 s), whereas the experts performed best with fluoroscopy (mean: 59 s). iPad assistance lowered radiation exposure by a factor of 3 for the inexperienced physician and by a factor of 1.8 for the experts in comparison with fluoroscopy usage. We achieve a mean visualization accuracy of 2.5 mm.The proposed tablet computer-based AR system has proven helpful in assisting percutaneous interventions such as PCNL and shows benefits compared to other state-of-the-art assistance systems. A drawback of the system in its current state is the lack of depth information. Despite that, the simple integration into the clinical workflow highlights the potential impact of this approach to such interventions.

[31]	STATE A, HIROTA G, CHEN D T, et al. Superior augmented reality registration by integrating landmark tracking and magnetic tracking[C]// Proceedings of the 23rd annual conference on computer graphics and interactive techniques. 1996: 429-438. 本文引用 [1]

[32]	MARCHAND E, UCHIYAMA H, SPINDLER F. Pose estimation for augmented reality: a hands-on survey[J]. IEEE transactions on visualization and computer graphics, 2015, 22(12):2633-2651. 本文引用 [1]

[33]

QIN

, LIU

, YU

, et al. π-ba:Bundle adjustment acceleration on embedded FPGAs with co-observation optimization[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM).IEEE,2019: 100-108.

本文引用 [3]

[34]	LIU Q, WAN Z, YU B, et al. An energy-efficient and runtime-reconfigurable fpga-based accelerator for robotic localization systems[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2022:1-2. 本文引用 [2]

[35]	SUN R, LIU P, XUE J, et al. Bax:A bundle adjustment accelerator with decoupled access/execute architecture for visual odometry[J]. IEEE Access, 2020(8):75530-75542. 本文引用 [2]

[36]

SON

, CHOI

K S

, KIM

. Design of the Bundle Adjustment FPGA-SoC Architecture for Real Time Vision Based SLAM in AR Glasses[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE 21st International Conference on Control,Automation and Systems (ICCAS).IEEE,2021: 2152-2155.

本文引用 [2]

[37]	GUO R, NG M L, TANG H T, et al. An FPGA-based Accelerator for Local Bundle Adjustment of Stereo Visual Odometry[C]// TENCON 2022-2022 IEEE Region 10 Conference (TENCON).IEEE.IEEE,2022: 1-7. 本文引用 [2]

[38]

成祥, 陈迟晓, 翟鹏, 等. 面向光束平差法的视觉 SLAM 并行计算架构与实现[J]. 小型微型计算机系统, 2022, 43(9):1931-1938.

CHENG

, CHEN

CH X

, ZHAI

, et al. Visual SLAM Parallel Computing Architecture and Implementation for Beam Adjustment Method[J]. Journal of Chinese Mini-Micro Computer Systems, 2022, 43 (9):1931-1938 (in Chinese).

本文引用 [2]

[39]

LIU

, HUANG

, ZHAO

, et al. BOHA:A high performance VSLAM backend optimization hardware accelerator using recursive fine-grain H-matrix decomposition and early-computing with approximate linear solver[J]. IEEE Transactions on Circuits and Systems II:Express Briefs, 2023, 70(10):3827-3831.

本文引用 [2]

[40]

NIAN

, ZHANG

, NIU

, et al. Energy-Efficient Pose-Estimation FPGA-Accelerator for Real-Time Mobile V-SLAM Robot[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE IEEE 66th International Midwest Symposium on Circuits and Systems (MWSCAS).IEEE,2023: 566-570.

本文引用 [2]

[41]	NIU L, ZHANG W, NIAN C, et al. Hardware-Software Co-Design of Matrix-Solving for Non-Linear Optimization in SLAM Systems[C]// IECON 2023-49th Annual Conference of the IEEE Industrial Electronics Society.IEEE,2023:1-8. 本文引用 [4]

[42]	LI Z, CHEN Y, GONG L, et al. An 879gops 243mw 80fps vga fully visual cnn-slam processor for wide-range autonomous exploration[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2019:134-136. 本文引用 [3]

[43]	BURRI M, NIKOLIC J, GOHL P, et al. The EuRoC micro aerial vehicle datasets[J]. The International Journal of Robotics Research, 2016, 35(10):1157-1163. 本文引用 [1]

[44]	WANG Q, HU A, HAN D, et al. Hardware Accelerator Design of Non-linear Optimization Correlative Scan Matching Algorithm in 2D LiDAR SLAM for Mobile Robots[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2022:89-93. 本文引用 [2]

[45]	JIN W, ZENG X, JING M, et al. An Efficient Bundle Adjustment Approach for Stereo Visual Odometry with Pose Consensus[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2023:1-6. 本文引用 [1]

[46]	LI M, ZHU H, HE S, et al. SLAM-CIM:A Visual SLAM Backend Processor With Dynamic-Range-Driven-Skipping Linear-Solving FP-CIM Macros[J]. IEEE Journal of Solid-State Circuits, 2024. 本文引用 [3]

[47]	FASOGBON P, HEIKKILÄ L, AKSU E. Frame selection to accelerate Depth from Small Motion on smartphones[C]// IECON 2019-45th Annual Conference of the IEEE Industrial Electronics Society.IEEE, 2019,1:113-118. 本文引用 [1]

[48]	LU W, XIANG Z, LIU J. High-performance visual odometry with two-stage local binocular BA and GPU[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2013:1107-1112. 本文引用 [1]

[49]	GOPINATH S, DANTU K, KO S Y. Improving the Performance of Local Bundle Adjustment for Visual-Inertial SLAM with Efficient Use of GPU Resources[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2023:6239-6245. 本文引用 [1]

[50]	MASMANO M, RIPOLL I, CRESPO A, et al. TLSF:A new dynamic memory allocator for real-time systems[C]// Proceedings. 16th Euromicro Conference on Real-Time Systems,2004:79-88. 本文引用 [1]

[51]	FENG Y S, CHEN J Y, WANG H C, et al. Learning-Based Gaussian Belief Propagation for Bundle Adjustment in Visual SLAM[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2022:166-171. 本文引用 [1]

[52]	WU C, AGARWAL S, CURLESS B, et al. Multicore bundle adjustment[C]// CVPR 2011.IEEE,2011:3057-3064. 本文引用 [1]

[53]	LIN M, CAO Q,ZHANGH. PVO:Panoramic visual odometry[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE 3rd International Conference on Advanced Robotics and Mechatronics (ICARM).IEEE,2018: 491-496. 本文引用 [1]

[54]

万泽宇, 丁朝阳, 王怡阳, 等. 同步定位与建图技术的发展现状[J]. 微纳电子与智能制造, 2022, 4(2):22-45.DOI:10.19816/j.cnki.10-1594/tn.2022.02.022.

WAN

Z Y

, DING

CH Y

, WANG

Y Y

, et al. Development status of synchronous positioning and mapping technology[J]. Micro Nano Electronics and Intelligent Manufacturing, 2022, 4 (2):22-45.DOI:10.19816/j.cnki.10-1594/tn.2022.02.022 (in Chinese).

本文引用 [1]

[55]	ZHANG W, NIU L, DING C, et al. ROPY-SLAM:a Heterogeneous CPU-FPGA System for Simultaneous Localization and Mapping[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2022:125-126. 本文引用 [1]

[56]	WANG J, ZHAN Y, WANG Z, et al. A reconfigurable matrix multiplication coprocessor with high area and energy efficiency for visual intelligent and autonomous mobile robots[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2021:1-3. 本文引用 [1]

[57]	CAMPOS C, ELVIRA R, RODRÍGUEZ J J G, et al. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam[J]. IEEE Transactions on Robotics, 2021, 37(6):1874-1890. 本文引用 [1]

[58]	LIU Y, LI J, HUANG K, et al. MobileSP:An FPGA-based real-time keypoint extraction hardware accelerator for mobile VSLAM[J]. IEEE Transactions on Circuits and Systems I:Regular Papers, 2022, 69(12):4919-4929.

[59]	XU Z, YU J, YU C, et al. CNN-based Feature-point Extraction for Real-time Visual SLAM on Embedded FPGA[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE IEEE 28th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM).IEEE,2020: 33-37.

[60]

《中国公路学报》编辑部. 中国汽车工程学术研究综述·2023[J]. 中国公路学报, 2023, 36(11):1-192.DOI:10.19721/j.cnki.1001-7372.2023.11.001.

摘要

为进一步提升中国汽车工程学科实力，促进中国汽车技术发展，从汽车电动化与节能、汽车智能化与网联化、汽车动力学与控制、汽车噪声-振动-声振粗糙度（Noise，Vibration，Harshness，NVH）控制与轻量化、汽车电子电气与软件技术、汽车测试与评价技术6个方面，系统梳理了国内外汽车工程领域近年的学术研究现状、前沿热点问题、最新研究成果及未来发展前景。汽车电动化与节能梳理了纯电动汽车、插电式混合动力汽车、氢燃料电池汽车、增程式电动汽车和节能汽车的关键技术研究；汽车智能化与网联化梳理了智能驾驶环境感知技术、自动驾驶定位技术、智能车决策规划、运动控制技术、车路协同、智能安全技术、车联网安全技术和智能座舱与人机交互等研究；汽车动力学与控制梳理了线控制动技术、线控转向技术、线控悬架技术和线控底盘协同控制技术研究；汽车NVH控制与轻量化梳理了汽车气动噪声预测与优化、纯电动汽车系统的NVH控制、声学超材料与汽车结构振动控制、汽车噪声主动控制、汽车轻量化技术以及碰撞安全技术等研究；汽车电子电气与软件技术梳理了汽车电子电气架构、汽车软件技术与空中下载技术（Over the Air，OTA）升级、芯片与系统功能集成等研究；汽车测试与评价技术梳理了燃油车、新能源汽车和智能网联汽车的测试与评价技术等研究。该综述可为中国汽车工程学科研究的进一步发展提供研究参考，为汽车工程关键技术发展提供方向引导。

Editorial Department of China Journal of Highways. Academic Research Review of China Automotive Engineering·2023[J]. China Journal of Highways, 2023, 36(11):1-192.DOI: 10.19721/j.cnki.1001-7372.2023.11.001 (in Chinese).

本文引用 [1]

[61]	WAN Z, YU B, LI T Y, et al. A survey of fpga-based robotic computing[J]. IEEE Circuits and Systems Magazine, 2021, 21(2):48-74. 本文引用 [2]

[62]	KONOLIGE K, AGRAWAL M. FrameSLAM:From bundle adjustment to real-time visual mapping[J]. IEEE Transactions on Robotics, 2008, 24(5):1066-1077. 本文引用 [1]

[63]	ZHANG W, ZHANG C, NIU L, et al. An Efficient FPGA Design for Fixed-point Exponential Calculation[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2022:44-45. 本文引用 [1]

[64]	HAN G, ZHANG W, NIU L, et al. Hardware Implementation of Approximate Fixed-point Divider for Machine Learning Optimization Algorithm[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2022:22-25. 本文引用 [1]

[65]

管开轩, 余江, 周建中, 等. 高水平科技自立自强下我国集成电路人才培养“痛点”与对策[J]. 中国科学院院刊, 2023, 38(2):324-332.

GUAN

K X

, YU

, ZHOU

J ZH

, et al. "Pain Points" and Countermeasures for China's IC Talent Training under the Self reliance of High level Science and Technology[J]. Journal of the Chinese Academy of Sciences, 2023, 38(2):324-332 (in Chinese).

本文引用 [1]

[66]	O'LOUGHLIN D, COFFEY A, CALLALY F, et al. Xilinx vivado high level synthesis:Case studies[C]// Irish Signals & Systems Conference & China-ireland International Conference on Information & Communications Technologies.IET, 2014.DOI:10.1049/cp.2014.0713. 本文引用 [1]