间接飞行时间图像传感器技术综述

徐江涛; 陈全民; 王欢欢; 聂凯明; 高静

doi:10.20193/j.ices2097-4191.2024.05.001

PDF(3980 KB)

集成电路与嵌入式系统 ›› 2024, Vol. 24 ›› Issue (5) : 1-9. DOI: 10.20193/j.ices2097-4191.2024.05.001

CMOS图像传感器研究专栏

间接飞行时间图像传感器技术综述

作者信息 +

Overview of indirect time-of-flight image sensor technology

Author information +

文章历史 +

摘要

基于飞行时间(Time of Flight, ToF)的三维(three dimensional, 3D)成像技术属于主动式三维成像技术,其通过测量调制光“飞行”往返于目标物体与传感器之间所需要的时间,进而计算出目标物体的距离信息。相较于其他3D成像方式,基于ToF的3D成像方法具有微型化、结构简单、功耗低等优势。随着技术的发展,间接ToF(Indirect ToF, IToF)图像传感器像素尺寸逐渐缩小、分辨率逐渐增大、精度逐渐提高,应用于多种场景,但是其仍然存在背景光干扰、多径干扰、运动伪影等问题。本文第一节介绍ToF图像传感器的工作原理;第二节总结分析ToF图像传感器的参数指标及其发展趋势;第三节分析ToF图像传感器面临的挑战并提供解决方案;第四节分析应用于ToF技术的图像校正及还原算法。

Abstract

Three-dimensional (3D) imaging technology based on Time-of-Flight (ToF) belongs to active 3D imaging technology.It calculates the distance information of the target object by measuring the time needed for modulated light to "fly" back and forth between the target object and the sensor.Compared to other 3D imaging methods, ToF-based 3D imaging methods have significant advantages such as miniaturization, simple structure,and low power consumption.With the development of technology,Indirect Time-of-Flight (IToF) image sensor pixel size gradually decreases,resolution increases,and accuracy improves,making it applicable in various scenarios.However,it still faces challenges such as background light interference,multipath interference,and motion artifacts.The working principle of ToF image sensors is introduced in section 1.The parameter indicators of ToF image sensors and their development trends are analyzed in section 2.The challenges faced by ToF image sensors are analyzed and the solutions are proposed in section 3.The image correction and restoration algorithms applied to ToF technology are introduced in section 4.

导出引用

徐江涛, 陈全民, 王欢欢, 等. 间接飞行时间图像传感器技术综述[J]. 集成电路与嵌入式系统. 2024, 24(5): 1-9 https://doi.org/10.20193/j.ices2097-4191.2024.05.001

XU Jiangtao, CHEN Quanmin, WANG Huanhuan, et al. Overview of indirect time-of-flight image sensor technology[J]. Integrated Circuits and Embedded Systems. 2024, 24(5): 1-9 https://doi.org/10.20193/j.ices2097-4191.2024.05.001

中图分类号： TN401 (理论)

参考文献

列表( 原文顺序 | 文献年度倒序 | 文中引用次数倒序 ) 可视化分析

[1]	GENG J. Structured-light 3D surface imaging:A tutorial[J]. Advances in Optics and Photonics, 2011, 2(2):128-160. 本文引用 [1]

[2]	张安. 基于SPAD的飞行时间三维图像传感器关键技术研究[D]. 天津: 天津大学, 2018. ZHANG AN. Key Technology Research on SPAD Based Flight Time 3D Image Sensor[D]. Tianjin: Tianjin University, 2018. (in Chinese) 本文引用 [1]

[3]	史晓琳. 基于飞行时间的3D图像传感器关键技术研究[D]. 天津大学, 2020. SHI X L. Research on Key Technologies of 3D Image Sensors Based on Flight Time[D]. Tianjin: Tianjin University, 2020. (in Chinese) 本文引用 [1]

[4]	BELLISAI S, VILLA F, TISA S, et al. Indirect time-of-flight 3D ranging based on SPADs[C]// Quantum Sensing and Nanophotonic Devices IX. SPIE, 2012, 8268:282-289. 本文引用 [1]

[5]	PAYNE A D, JONGENELEN A P P, DORRINGTON A A, et al. Multiple frequency range imaging to remove measurement ambiguity[C]// Optical 3-d measurement techniques, 2009. 本文引用 [1]

[6]	NICLASS C, FAVI C, KLUTER T, et al. Single-photon synchronous detection[J]. IEEE Journal of Solid-State Circuits, 2009, 44(7):1977-1989. 本文引用 [1]

[7]	HAFIANE M L, WAGNER W, DIBI Z, et al. Analysis and estimation of NEP and DR in CMOS TOF-3D image sensor based on MDSI[J]. Sensors and Actuators A:Physical, 2011, 169(1): 66-73. 本文引用 [1]

[8]	BAMJI C S, MEHTA S, THOMPSON B, et al. IMpixel 65nm BSI 320MHz demodulated TOF Image sensor with 3μm global shutter pixels and analog binning[C]// 2018 IEEE International Solid-State Circuits Conference(ISSCC).IEEE, 2018:94-96. 本文引用 [2]

[9]	EBIKO Y, YAMAGISHI H, TATANI K, et al. Low power consumption and high resolution 1280×960 Gate Assisted Photonic Demodulator pixel for indirect Time of flight[C]// 2020 IEEE International Electron Devices Meeting (IEDM).IEEE, 2020:33.1.1-33.1.4. 本文引用 [2]

[10]	KEEL M S, KIM D, KIM Y, et al. A 4-tap 3.5 μm 1.2 Mpixel indirect time-of-flight CMOS image sensor with peak current mitigation and multi-user interference cancellation[C]// 2021 IEEE International Solid-State Circuits Conference (ISSCC).IEEE, 2021, 64:106-108.

[11]

BELLISAI

, BRONZI

, VILLA

F A

, et al. Single-photon pulsed-light indirect time-of-flight 3D ranging[J]. Optics express, 2013, 21(4):5086-5098.

https://doi.org/10.1364/OE.21.005086

https://www.ncbi.nlm.nih.gov/pubmed/23482043

本文引用 [1] 摘要

"Indirect" time-of-flight is one technique to obtain depth-resolved images through active illumination that is becoming more popular in the recent years. Several methods and light timing patterns are used nowadays, aimed at improving measurement precision with smarter algorithms, while using less and less light power. Purpose of this work is to present an indirect time-of-flight imaging camera based on pulsed-light active illumination and a 32 × 32 single-photon avalanche diode array with an improved illumination timing pattern, able to increase depth resolution and to reach single-photon level sensitivity.

[12]	沈奇峰. 巷道视觉重建与设备碰撞预警技术研究[D]. 西安: 西安科技大学, 2022.DOI:10.27397/d.cnki.gxaku.2022.000470. SHEN Q F. Research on Lane Visual Reconstruction and Equipment Collision Warning Technology[D]. Xi'an: Xi'an University of Science and Technology, 2022.DOI:10.27397/d.cnki.gxaku.2022-000470. (in Chinese) 本文引用 [1]

[13]	XIN Y, ZHANG B, HU C, et al. A 320× 240 I-ToF CMOS image sensor with 2-tap 5.6 m pixel and mismatch-nonlinearity suppression[C]// 2021 IEEE International Symposium on Circuits and Systems (ISCAS).IEEE, 2021:1-4. 本文引用 [1]

[14]	XIN Y, ZHANG B, Hu C, et al. A Low Power i-ToF LiDAR With Nonlinearity Self-calibration Technique[J]. IEEE Transactions on Instrumentation and Measurement, 2023. 本文引用 [1]

[15]	KUO C C, KURODA R. A 134×132 4-Tap CMOS Indirect Time-of-Flight Range Imager Using In-Pixel Memory Array With 10 Kfps High-Speed Mode and High Precision Mode[J]. IEEE Journal of Solid-State Circuits, 2023. 本文引用 [1]

[16]	PIRON F, MORRISON D, YUCE M R, et al. A review of single-photon avalanche diode time-of-flight imaging sensor arrays[J]. IEEE Sensors Journal, 2020, 21(11):12654-12666. 本文引用 [1]

[17]	张澍尧, 钱先云, 樊振凯, 等. 机器视觉技术在空中加油中的应用研究[C]// 第五届中国航空科学技术大会论文集, 2021. ZHANG SH Y, QIAN X Y, FAN ZH K, et al. Application of Machine Vision in Aerial Refueling[C]// Proceedings of the 5th China Aviation Science and Technology Conference, 2021. (in Chinese) 本文引用 [1]

[18]	YU SH ZH. A CMOS Time-of-Flight Image Sensor with High Dynamic Range Digital Pixel[C]// 2021 IEEE 14th International Conference on ASIC (ASICON), 2021.

[19]	DIELACHER M, FLATSCHER M, GABL R, et al. Advancements in indirect Time of Flight image sensors in front side illuminated CMOS[C]// ESSDERC 2021-IEEE 51st European Solid-State Device Research Conference (ESSDERC).IEEE, 2021:139-142.

[20]	RINGBECK T, MÖLLER T, HAGEBEUKER B. Multidimensional measurement by using 3-D PMD sensors[J]. Advances in Radio Science, 2007(5):135-146. 本文引用 [1]

[21]	BAMJI C S, O'CONNOR P, ELKHATIB T, et al. A 0.13 μm CMOS system-on-chip for a 512×424 time-of-flight image sensor with multi-frequency photo-demodulation up to 130 MHz and 2 GS/s ADC[J]. IEEE Journal of Solid-State Circuits, 2014, 50(1):303-31. 本文引用 [1]

[22]	LINDNER M, KOLB A. Compensation of motion artifacts for time-of-flight cameras[C]// Workshop on Dynamic 3D Imaging. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009:16-27. 本文引用 [1]

[23]	KATO Y, SANO T, MORIYAMA Y, et al. 320×240 Back-Illuminated 10-μm CAPD Pixels for High-Speed Modulation Time-of-Flight CMOS Image Sensor[J]. IEEE Journal of Solid-State Circuits, 2018, 53(4):1071-1078. 本文引用 [1]

[24]	KWON Y, SEO S, CHO S, et al. A 2.8 μm pixel for time of flight CMOS image sensor with 20 ke-full-well capacity in a tap and 36% quantum efficiency at 940 nm wavelength[C]// 2020 IEEE International Electron Devices Meeting (IEDM).IEEE, 2020:33.2.1-33.2.4. 本文引用 [1]

[25]	KEEL M S, KIM D, KIM Y, et al. A 1.2-m pixel indirect time-of-flight image sensor with 4-tap 3.5-μm pixels for peak current mitigation and multi-user interference cancellation[J]. IEEE Journal of Solid-State Circuits, 2021, 56(11):3209-3219. 本文引用 [1]

[26]	KEEL M S, JIN Y G, KIM Y, et al. A 640×480 indirect time-of-flight CMOS image sensor with 4-tap 7-μm global-shutter pixel and fixed-pattern phase noise self-compensation scheme[C]// 2019 Symposium on VLSI Circuits.IEEE, 2019:C258-C259.

[27]	SEURIN J F, XU G, KHALFIN V, et al. Progress in high-power high-efficiency VCSEL arrays[C]// Vertical-Cavity Surface-Emitting Lasers XIII.SPIE, 2009, 7229:19-29. 本文引用 [1]

[28]	WARREN M E, PODVA D, DACHA P, et al. Low-divergence high-power VCSEL arrays for lidar application[C]// Vertical-Cavity Surface-Emitting Lasers XXII.SPIE, 2018, 10552:72-81. 本文引用 [1]

[29]	Noraky J. Algorithms and systems for low power time-of-flight imaging[C]// 2019 IEEE International Conference on Image Processing (ICIP).IEEE, 2019:3023-3024. 本文引用 [1]

[30]	FREEDMAN D, SMOLIN Y, KRUPKA E, et al. SRA: Fast removal of general multipath for ToF sensors[C]// Computer Vision-ECCV 2014:13th European Conference,Zurich,Switzerland,September 6-12,2014,Proceedings,Part I 13.Springer International Publishing, 2014:234-249. 本文引用 [1]

[31]	SHAHANDASHTI P F, LÓPEZ P, BREA V M, et al. Fast Time-Domain Super-Resolution for Single-Shot Multi-Path ToF Imaging[C]// 2022 29th IEEE International Conference on Electronics,Circuits and Systems (ICECS).IEEE, 2022:1-4. 本文引用 [1]

[32]	BHANDARI A, FEIGIN M, IZADI S, et al. Resolving multipath interference in kinect:An inverse problem approach[C]// SENSORS,2014 IEEE.IEEE, 2014:614-617. 本文引用 [1]

[33]	LIM Y S, LEE S H, KWON W H, et al. Depth image super resolution method for time-of-flight camera based on machine learning[C]// AI and Optical Data Sciences III.SPIE, 2022, 12019:191-196. 本文引用 [2]

[34]	SON K, LIU M Y, TAGUCHI Y. Learning to remove multipath distortions in time-of-flight range images for a robotic arm setup[C]// 2016 IEEE International Conference on Robotics and Automation (ICRA).IEEE, 2016:3390-3397. 本文引用 [1]

[35]	MARCO J, HERNANDEZ Q, MUNOZ A, et al. DeeptToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging[J]. ACM Transactions on Graphics (ToG), 2017, 36(6):1-12. 本文引用 [1]

[36]	AGRESTI G, ZANUTTIGH P. Deep learning for multi-path error removal in ToF sensors[C]// Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018. 本文引用 [2]

[37]	GUO Q, FROSIO I, GALLO O, et al. Tackling 3d tof artifacts through learning and the flat dataset[C]// Proceedings of the European Conference on Computer Vision (ECCV), 2018:368-383. 本文引用 [1]

[38]	WANG T L, AO L, ZHENG J, et al. Reconstructing Depth Images for Time-of-Flight Cameras Based on Second-Order Correlation Functions[C]// Photonics. MDPI, 2023, 10(11):1223. 本文引用 [1]

[39]	WEI SIYUAN. A Real-Time 2D/3D Perception Visual Vector Processor for 1920×1080 High-Resolution High-Speed Intelligent Vision Chips[J]. IEEE Transactions on Circuits and Systems I:Regular Papers (2023). 本文引用 [1]

[40]	SCHMIDT M, JÄHNE B. Efficient and robust reduction of motion artifacts for 3d time-of-flight cameras[C]// 2011 International Conference on 3D Imaging (IC3D).IEEE, 2011:1-8. 本文引用 [2]

[41]	LEE S. Time-of-flight depth camera motion blur detection and deblurring[J]. IEEE Signal Processing Letters, 2014, 21(6):663-666. 本文引用 [2]

[42]	WACH H, DOWSKI JR E R. Noise modeling for design and simulation of computational imaging systems[C]// Visual Information Processing XIII.SPIE, 2004, 5438:159-170. 本文引用 [1]

[43]	KANG J, PARK Y, H WANG J H, et al. An Indirect Time-of-Flight Sensor With Tetra-Pixel Architecture Calibrating Tap Mismatch in a Single Frame[J]. IEEE Solid-State Circuits Letters, 2022(5):284-287. 本文引用 [1]

[44]	ZHANG B, HU C, LAI J, et al. A two-tap indirect time-of-flight CMOS image sensor with pump gate modulator for low-power applications[J]. IEEE Transactions on Electron Devices, 2021, 69(6):2858-2864. 本文引用 [1]

[45]	PIAO C, AHN Y, KIM D, et al. A Low-Power Indirect Time-of-Flight CMOS Image Sensor With Fixed Depth Noise Compensation and Dual-Mode Imaging for Depth Dynamic Range Enhancement[J]. IEEE Transactions on Circuits and Systems I:Regular Papers, 2022, 69(10):3989-3999. 本文引用 [1]

[46]	VOLAK J, KONIAR D, JABLONCIK F, et al. Interference artifacts suppression in systems with multiple depth cameras[C]// 2019 42nd International Conference on Telecommunications and Signal Processing (TSP).IEEE, 2019:472-476.

[47]	KUHNERT K D, STOMMEL M. Fusion of stereo-camera and pmd-camera data for real-time suited precise 3d environment reconstruction[C]// 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.IEEE, 2006:4780-4785. 本文引用 [3]

[48]	SCHILLER I, BEDER C, KOCH R. Calibration of a PMD-camera using a planar calibration pattern together with a multi-camera setup[J]. The international archives of the photogrammetry,remote sensing and spatial information sciences, 2008(21):297-302. 本文引用 [2]

[49]	J JUNG, J LEE, Y JEONG, et al. Time-of-flight sensor calibration for a color and depth camera pair[J]. IEEE Trans. Pattern Anal. Mach. Intell., 2015, 37(7):1501-1513. 本文引用 [4]

[50]	SHRESTHA S, HEIDE F, HEIDRICH W, et al. Computational imaging with multi-camera time-of-flight systems[J]. ACM Transactions on Graphics (ToG), 2016, 35(4):1-11. 本文引用 [1]

[51]	HUANG T, QIAN K, LI Y. All Pixels Calibration for ToF Camera[C]// IOP Conference Series:Earth and Environmental Science.IOP Publishing, 2018, 170(2):022164. 本文引用 [1]

[52]	K UZNETSOVA A, ROSENHAHN B. On calibration of a low-cost time-of-flight camera[C]// Computer Vision-ECCV 2014 Workshops:Zurich,Switzerland,September 6-7 and 12,2014,Proceedings,Part I 13. Springer International Publishing, 2015:415-427. 本文引用 [1]

[53]	RODRIGUEZ B, ZHANG X, RAJAN D. Probabilistic Modeling of Multicamera Interference for Time-of-Flight Sensors[J]. Sensors, 2023, 23(19):8047.

[54]	FALIE D, BUZULOIU V. Distance errors correction for the time of flight (ToF) cameras[C]// 2008 IEEE International Workshop on Imaging Systems and Techniques.IEEE, 2008:123-126.

[55]	FÜRSATTEL P, PLACHT S, BALDA M, et al. A comparative error analysis of current time-of-flight sensors[J]. IEEE Transactions on Imaging, 2015, 2(1):27-41.

[56]	REINHARDT A, BRADLEY C, HECHT A, et al. Windowed region-of-interest non-uniformity correction and range walk error correction of a 3D flash LiDAR camera[J]. Optical Engineering, 2021, 60(2):023103. 本文引用 [1]

[57]	ALI A K, ADIBI P, EHSANI M S. Depth Map Reconstruction and Enhancement With Local and Patch Manifold Regularized Deep Depth Priors[J]. IEEE Access, 2021(9):136111-136125. 本文引用 [1]

[58]	ERGÜN B, KURTAR G. Calibration of a Time-of-Flight Camera Using Probability and Reflection Analysis[J]. IEEE Sensors Journal, 2023. 本文引用 [1]

[59]	KIM D, LEE S, PARK D, et al. 5.4 A dynamic pseudo 4-tap CMOS time-of-flight image sensor with motion artifact suppression and background light cancelling over 120klux[C]// 2020 IEEE International Solid-State Circuits Conference(ISSCC).IEEE, 2020:100-102.

[60]	HATAKEYAMA K, OKUBO Y, NAKAGOME T, et al. A hybrid indirect ToF image sensor for long-range 3D depth measurement under high ambient light conditions[C]// 2022 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits).IEEE, 2022:46-47.

[61]	WEI S, NING K, KANG L, et al. A Real-Time 2D/3D Perception Visual Vector Processor for 1920×1080 High-Resolution High-Speed Intelligent Vision Chips[J]. IEEE Transactions on Circuits and Systems I:Regular Papers, 2023.

[62]	KEEL M S, JIN Y G, KIM Y, et al. A VGA Indirect Time-of-Flight CMOS Image Sensor With 4-Tap 7-mu m Global-Shutter Pixel and Fixed-Pattern Phase Noise Self-Compensation[J]. IEEE Journal of Solid-State Circuits, 2019, 55(4):889-897.

[63]	YU S, ZHANG Y, ZHOU F, et al. A CMOS time-of-flight image sensor with high dynamic range digital pixel[C]// 2021 IEEE 14th International Conference on ASIC (ASICON).IEEE, 2021:1-4.

[64]	MIYAZAWA R, SHIRAKAWA Y, MARS K, et al. A Time-of-Flight Image Sensor Using 8-Tap PN Junction Demodulator Pixels[J]. Sensors, 2023, 23(8):3987.

[65]	HSU T H, LIAO T, LEE N A, et al. A CMOS time-of-flight depth image sensor with in-pixel background light cancellation and phase shifting readout technique[J]. IEEE Journal of Solid-State Circuits, 2018, 53(10):2898-2905.

[66]	刘志强, 董杰, 马治强, 等. 一种消除模拟 TOF 阵列探测器 FPN 的两级 CDS 电路[J]. 微电子学, 2023, 53(1):95-101. LIU ZH Q, DONG J, MA ZH Q, et al. A two-stage CDS circuit for eliminating analog TOF array detector FPN[J]. Microelectronics, 2023, 53(1):95-101. (in Chinese)

[67]	KUO C C, KURODA R. A 4-Tap CMOS Time-of-Flight Image Sensor with In-pixel Analog Memory Array Achieving 10Kfps High-Speed Range Imaging and Depth Precision Enhancement[C]// 2022 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits).IEEE, 2022:48-49.