神经辐射场硬件加速器设计研究综述

吴立舟, 朱浩哲, 陈迟晓

集成电路与嵌入式系统 ›› 2024, Vol. 24 ›› Issue (11) : 41-50.

PDF(16666 KB)
PDF(16666 KB)
集成电路与嵌入式系统 ›› 2024, Vol. 24 ›› Issue (11) : 41-50. DOI: 10.20193/j.ices2097-4191.2024.0020
智能机器人高能效专用芯片研究专栏

神经辐射场硬件加速器设计研究综述

作者信息 +

Review on hardware accelerators design for neural radiation field

Author information +
文章历史 +

摘要

神经辐射场(NeRF)是一种用于重建三维场景的新兴方法,其在机器人领域的应用前景备受关注。NeRF通过多层感知机(MLP)学习三维场景特征,实现高保真的图像渲染,并为机器人在复杂环境中的导航、定位和感知提供基础。其核心流程包括光线采样、特征提取和体渲染,具有计算量大和非规则存储访问密集的特点,限制了在现有硬件平台,尤其是端侧设备上的部署,亟需探索新的硬件架构和软硬件协同优化方案。本文系统阐述了NeRF的技术原理与算法演进,并探讨了其在现有硬件设备上的性能瓶颈。在此基础上,详细介绍了经典的NeRF硬件加速器工作,归纳出图像相似性优化、空间稀疏性优化、存储器访问优化三种主要优化方向,并分析了不同工作技术的共性与差异。此外,结合SLAM、AIGC等应用场景,探讨了当前NeRF加速器在处理开放场景任务时所面临的可扩展性和存储限制方面的技术局限和挑战。最后,提出未来发展的建议,以期为NeRF硬件加速器的进一步应用和优化提供启发。

Abstract

Neural Radiance Fields (NeRF) is an emerging method for reconstructing 3D scenes, garnering significant attention for its potential applications in the field of robotics. NeRF uses Multi-Layer Perceptrons (MLPs) to learn 3D scene features, achieving high-fidelity image rendering and providing a foundation for navigation, localization, and perception in complex environments. Its core processes, including ray sampling, feature extraction, and volumetric rendering, are computationally intensive and involve irregular memory access patterns, which limits deployment on existing hardware platforms, especially edge devices. To advance the practical application of NeRF technology, new hardware architectures and solutions for co-optimization of hardware and software are necessary. This review systematically elucidates the principles and evolution of NeRF technology, exploring the performance bottlenecks encountered during its hardware execution. The review provides a detailed review of classic NeRF hardware accelerators, summarizing three main optimization directions: image similarity optimization, spatial sparsity optimization, and memory access optimization, and analyzes the commonalities and differences among various techniques. Additionally, the review examines the technical limitations and challenges of current NeRF accelerators in handling open scene tasks, considering applications such as SLAM and AIGC, particularly in terms of scalability and storage constraints. Finally, the review offers suggestions for future development to inspire further applications and optimization of NeRF hardware accelerators.

关键词

神经辐射场 / 3D重建 / 图像渲染 / 加速器 / TensoRF

Key words

NeRF / 3D reconstruction / graphic rendering / accelerator / TensoRF

引用本文

导出引用
吴立舟, 朱浩哲, 陈迟晓. 神经辐射场硬件加速器设计研究综述[J]. 集成电路与嵌入式系统. 2024, 24(11): 41-50 https://doi.org/10.20193/j.ices2097-4191.2024.0020
WU Lizhou, ZHU Haozhe, CHEN Chixiao. Review on hardware accelerators design for neural radiation field[J]. Integrated Circuits and Embedded Systems. 2024, 24(11): 41-50 https://doi.org/10.20193/j.ices2097-4191.2024.0020
中图分类号: TN492 (专用集成电路)   

参考文献

[1]
QUEST M. Introducing horizon workrooms: Remote collaboration reimagined[EB/OL].(2021-08-19)[2024-07-15].https://about.fb.com/news/2021/08/introducing-horizon-workrooms-remote-collaboration-reimagined/.
[2]
MIAN A S, BENNAMOUN M, OWENS R A. Automatic correspondence for 3D modeling: An extensive review[J]. International Journal of Shape Modeling, 2005, 11(2):253-291.
[3]
SCHROEDER W J, ZARGE J A, LORENSEN W E. Decimation of triangle meshes[C]// Proceedings of the 19th annual conference on computer graphics and interactive techniques,1992:65-70.
[4]
MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. Nerf: Representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2021, 65(1):99-106.
[5]
PINKUS A. Approximation theory of the MLP model in neural networks[J]. Acta numerica, 1999(8):143-195.
[6]
李晗. 基于DIBR视点合成的空洞填充方法[J]. 现代计算机, 2019(1):76-79.
LI H. Cavity filling method based on DIBR view synthesis[J]. Modern Computer, 2019(1):76-79 (in Chinese).
[7]
MAX N. Optical models for direct volume rendering[J]. IEEE Transactions on Visualization and Computer Graphics, 1995, 1(2):99-108.
[8]
YU A, LI R, TANCIK M, et al. Plenoctrees for real-time rendering of neural radiance fields[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision,2021:5752-5761.
[9]
GARBIN S J, KOWALSKI M, JOHNSON M, et al. Fastnerf:High-fidelity neural rendering at 200 fps[C]// Proceedings of the IEEE/CVF international conference on computer vision,2021.
[10]
REISER C, PENG S, LIAO Y, et al. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps[C]// Proceedings of the IEEE/CVF international conference on computer vision,2021:14335-14345.
[11]
CHEN A, XU Z, GEIGER A, et al. Tensorf: Tensorial radiance fields[C]// European conference on computer vision. Cham: Springer Nature Switzerland, 2022.
[12]
MÜLLER T, EVANS A, SCHIED C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM transactions on graphics (TOG), 2022, 41(4):1-15.
[13]
LI W, KLAUS MUELLER, ARIEKAUFMAN. Empty space skipping and occlusion clipping for texture-based volume rendering[C]// IEEE Visualization, 2003.
[14]
LI C, LI S, ZHAO Y, et al. Rt-nerf:Real-time on-device neural radiance fields towards immersive AR/VR rendering[C]// Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design, 2022:1-9.
[15]
LEE J, CHOI K, LEE J, et al. Neurex:A case for neural rendering acceleration[C]// Proceedings of the 50th Annual International Symposium on Computer Architecture,2023:1-13.
[16]
HAN D, RYU J,KIMS, et al. MetaVrain:A Mobile Neural 3-D Rendering Processor With Bundle-Frame-Familiarity-Based NeRF Acceleration and Hybrid DNN Computing[J]. IEEE Journal of Solid-State Circuits, 2023.
[17]
RYU J, KWON H, PARK W, et al. 20.7 NeuGPU:A 18.5 mJ/Iter Neural-Graphics Processing Unit for Instant-Modeling and Real-Time Rendering with Segmented-Hashing Architecture[C]// 2024 IEEE International Solid-State Circuits Conference (ISSCC).IEEE, 2024,67:372-374.
[18]
JO W, KIM S, LEE J, et al. NeRPIM:A 4.2 mJ/frame neural rendering processing-in-memory processor with space encoding block-wise mapping for mobile devices[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE,2023:1-2.
[19]
LI S, LI C, ZHU W, et al. Instant-3d:Instant neural radiance field training towards on-device AR/VR 3d reconstruction[C]// Proceedings of the 50th Annual International Symposium on Computer Architecture,2023:1-13.
[20]
ZHAO Y K, WU S, ZHANG J, et al. Instant-NeRF:Instant On-Device Neural Radiance Field Training via Algorithm-Accelerator Co-Designed Near-Memory Processing[C]// 2018 IEEE International Conference on Systems, Man, and Cybernetics SMC.IEEE 60th ACM/IEEE Design Automation Conference (DAC).IEEE,2023: 1-6.
[21]
SUCAR E, LIU S, ORTIZ J, et al. imap:Implicit mapping and positioning in real-time[C]// Proceedings of the IEEE/CVF international conference on computer vision,2021:6229-6238.
[22]
ZHU Z, PENG S, LARSSON V, et al. Nice-slam: Neural implicit scalable encoding for slam[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,2022:12786-12796.
[23]
WANG H, WANG J, AGAPITO L. Co-slam:Joint coordinate and sparse parametric encodings for neural real-time slam[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2023:13293-13302.
[24]
KOSIOREK A R, STRATHMANN H, ZORAN D, et al. Nerf-vae:A geometry aware 3d scene generative model[C]// International Conference on Machine Learning.PMLR,2021:5742-5752.
[25]
GU J, TREVITHICK A, LIN K E, et al. Nerfdiff:Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion[C]// International Conference on Machine Learning.PMLR,2023:11808-11826.
[26]
MENG Q, CHEN A, LUO H, et al. Gnerf:Gan-based neural radiance field without posed camera[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision,2021:6351-6361.
[27]
TANCIK M, CASSER V, YAN X, et al. Block-nerf: Scalable large scene neural view synthesis[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2022:8248-8258.
[28]
PUMAROLA A, CORONA E, PONS-MOLL G, et al. D-nerf:Neural radiance fields for dynamic scenes[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2021:10318-10327.
[29]
CHEN Y H, KRISHNA T, EMER J S, et al. Eyeriss:An energy-efficient reconfigurable accelerator for deep convolutional neural networks[J]. IEEE journal of solid-state circuits, 2016, 52(1):127-138.
[30]
JOUPPI N P, YOUNG C, PATIL N, et al. In-datacenter performance analysis of a tensor processing unit[C]// Proceedings of the 44th annual international symposium on computer architecture,2017:1-12.
[31]
RAO C, YU H, WAN H, et al. Icarus:A specialized architecture for neural radiance fields rendering[J]. ACM Transactions on Graphics (TOG), 2022, 41(6):1-14.

责任编辑: 薛士然
PDF(16666 KB)

Accesses

Citation

Detail

段落导航
相关文章

/