An irregular memory access prefetching framework based on enhanced reinforcement learning

Yang Yuze, Huang Zhengwei, Wang Yongwen

Integrated Circuits and Embedded Systems ›› 2026, Vol. 26 ›› Issue (8) : 1-18.

PDF(35936 KB)
PDF(35936 KB)
Integrated Circuits and Embedded Systems ›› 2026, Vol. 26 ›› Issue (8) : 1-18. DOI: 10.20193/j.ices2097-4191.2026.0006
Cover Articl

An irregular memory access prefetching framework based on enhanced reinforcement learning

Author information +
History +

Abstract

Irregular data access patterns in high-performance computing and intelligent computing often render traditional data prefetching techniques ineffective. Existing models that rely on fixed rules or offline learning based on specific program contexts also struggle to adapt to dynamically changing memory access patterns during runtime. While the Pythia reinforcement learning (RL) prefetching framework demonstrates adaptability through online learning, it still requires manual tuning under extreme irregular workloads, limiting its generalization in practical applications. This paper proposes IEP (Irregular Enhanced Pythia), a context-aware reinforcement learning prefetching framework to enhance the prediction capability for irregular memory access patterns. The framework introduces two key innovations. The first is an irregular feature enhancement module. It incorporates address bit masks and access sequence distance as state features to capture hidden spatiotemporal patterns in memory allocator behavior, thereby improving the representation of irregular memory accesses. The second is a hierarchical reward strategy module, which employs a dynamic reward mechanism combining confidence awareness and bandwidth sensitivity. This mechanism finely guides the agent’s learning process, accelerating policy optimization and improving final performance. Experiments were conducted using the ChampSim simulator, to test various irregular workloads. Results show that compared to the Pythia framework, the proposed solution achieves a maximum improvement of 2.27% in average prefetching accuracy and 2.90% in average single-core IPC for typical irregular workloads such as Ligra and PARSEC, while maintaining stable performance in multi-core environments.

Key words

irregular memory access / hardware prefetching / reinforcement learning / ChampSim / state features / dynamic reward mechanism

Cite this article

Download Citations
Yang Yuze , Huang Zhengwei , Wang Yongwen. An irregular memory access prefetching framework based on enhanced reinforcement learning[J]. Integrated Circuits and Embedded Systems. 2026, 26(8): 1-18 https://doi.org/10.20193/j.ices2097-4191.2026.0006

References

[1]
Wilkes M V. The memory wall and the CMOS end-point[J]. ACM SIGARCH Computer Architecture News, 1995, 23(4):4-6.
[2]
Machanick P. Approaches to addressing the memory wall[D]. Brisbane: University of Queensland, 2002.
[3]
Mittal S. A survey of recent prefetching techniques for processor caches[J]. ACM Computing Surveys (CSUR), 2016, 49(2):1-35.
[4]
宗鹏陈, 曲劭儒, 赵文哲, 等. 基于开源处理器的间接访问数据预取器设计[J]. 集成电路与嵌入式系统, 2026(1):47-53.DOI:10.20193/j.ices2097-4191.2025.0082.
Zong P C, Qu S R, Zhao W Z, et al. Design of an Indirect-Access Data Prefetcher Based on Open-Source Processors[J]. Integrated Circuits and Embedded Systems, 2026(1):47-53.DOI:10.20193/j.ices2097-4191.2025.0082 (in Chinese).
[5]
王锦涵, 李俊, 路冬冬, 等. 基于双倍步长数据流的硬件预取机制[J]. 计算机工程, 2019, 45(6):115-118,126.
Abstract
硬件数据预取技术可以有效提升处理器的访存性能,但传统流预取策略存在预取不及时的问题。为此,提出一种双倍步长流预取策略,并设计对应的预取部件结构。预取部件自动检测数据流的固定步长并将该步长扩大为原有的2倍,以计算预取地址。实验结果表明,加入该预取部件后,运行SPEC2006测试集的整数应用与浮点应用时,处理器性能最高可分别提升45%与57%,针对Cache Miss率较高的应用,该预取部件可以有效隐藏访存延时。
Wang J H, Li J, Lu D D, et al. Hardware Prefetching Mechanism Based on Double Stride Data Stream[J]. Computer Engineering, 2019, 45(6):115-118,126 (in Chinese).
[6]
Lotfi-kamran P, Sarbazi-azad H. Data Prefetching Techniques in Computer Systems[M]. Amsterdam: Elsevier, 2022.
[7]
Bhatia E, Chacon G, Pugsley S, et al. Perceptron-Based Prefetch Filtering[C]// ISCA, 2019.
[8]
Bera R, Kanellopoulos K, Nori A V, et al. Pythia:A Customizable Hardware Prefetching Framework Using Online Reinforcement Learning[C]// 54th Annual IEEE/ACM International Symposium on Microarchitecture, 2021.
[9]
Sutton R S, Barto A G. Reinforcement Learning:An Introduction[M]. Cambridge: MIT Press, 2017.
[10]
Zhang J X, Gu J. Multi-core helper thread prefetching for irregular data intensive applications(Article)[J]. Tongxin Xuebao/Journal on Communications, 2014, 35(8):137-146.
[11]
Kashyap I, Deb K. Grep: Performance Enhancement in MultiCore Processors using an Adaptive Graph Prefetcher[C]// 26th IEEE-Computer-Society Annual Symposium on VLSI (ISVLSI),2023.
[12]
王玉庆, 杨秋松, 李明树. 基于指令流混合模式学习的缓存预取算法[J]. 电子学报, 2023, 51(2):342-354.
Abstract
近期缓存预取算法的研究热点是使用基于模式识别的预测技术,例如Lookahead,推算访存请求的地址.此类算法一方面很难学习访存行为中的依赖缓存失效,另一方面不能精确控制预取请求发送和写回的时机.为了解决上述问题,本文提出了一种基于分支预测技术和混合模式学习的缓存预取(Instruction Flow Based Hybrid Prediction,IFBHP)算法.使用分支预测技术识别程序未来指令流中的访存指令流,通过多种地址关联模式的学习逐一计算访存指令流中每条指令的地址,写入访存地址队列.使用阈值评估未来指令流进入处理器主流水线的时刻,精确控制指令流所对应的预取请求的发送和写回.实验表明,本文算法相比STeMS(Spatio-Temporal Memory Streaming)算法、ISB++(Irregular Stream Buffer++)算法、SANGAM算法、IPCP(Instruction Pointer Classifier based spatial Prefetching)算法一级数据的读操作缓存失效次数分别平均减少31.58%,28.85%,17.85%,11.48%;本文算法相比STeMS算法、ISB++算法、SANGAM算法、IPCP算法一级数据的写操作缓存失效次数分别平均减少31.58%,28.85%,17.85%,11.48%.
Wang Y Q, Yang Q S, Li M S. Cache Prefetching Algorithm Based on Instruction Stream Hybrid Mode Learning[J]. Acta Electronica Sinica, 2023, 51(2):342-354 (in Chinese).
[13]
Bakhshalipour M, Shakerinava M, Lotfi-kamran P, et al.Bingo Spatial Data Prefetcher[C]// 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA),2019.
[14]
Fang J, Li J, Yang H, et al. AMPP: An Adaptive Multilayer Perceptron Prefetcher for Irregular Data Prefetching[C]// 2023 IEEE International Conference on High Performance Computing & Communications,Data Science & Systems, Smart City & Dependability in Sensor,Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), 2023.
[15]
Ipek E, Mutlu O, Martinez J F, et al. Self-Optimizing Memory Controllers:A Reinforcement Learning Approach[C]// ISCA, 2008.
[16]
Kao S C, Jeong G, Krishna T, et al. ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning[C]// MICRO, 2020.
[17]
Rummery G A, Niranjan M. On-line Q-learning using connectionist systems[R]. University of Cambridge,Department of Engineering Cambridge,UK, 1994.
[18]
La O G. 4-bit prefetch-type FCRAM having improved data write control circuit in memory cell array and method of masking data using the 4-bit prefetch-type FCRAM:US20030719306[P].2004-06-17.
[19]
Gober N, Chacon G, Wang L, et al. The Championship Simulator: Architectural Simulation for Education and Competition[EB/OL]. (2022-10-25)[2025-10-12]. https://arxiv.org/abs/2210.14324.
[20]
Doweck J, Kao W F, Lu A K Y, et al. Inside 6th-Generation Intel Core: New Microarchitecture Code-Named Skylake[J]. IEEE Micro, 2017, 37(2): 52-62.
[21]
Jimenez D A, Lin C. Dynamic Branch Prediction with Perceptrons[C]// HPCA, 2001.
[22]
Wu C J, Jaleel A, Hasenplaugh W, et al. SHiP:Signature-based Hit Predictor for High Performance Caching[C]// MICRO, 2011.
[23]
SPEC CPU 2006[EB/OL]. [2025-10-12]. https://www.spec.org/cpu2006/.
[24]
SPEC CPU 2017[EB/OL]. [2025-10-12]. https://www.spec.org/cpu2017/.
[25]
PARSEC[EB/OL].[2025-10-12]. http://parsec.cs.princeton.edu/.
[26]
Shun J, Blelloch G E. Ligra: A Lightweight Graph Processing Framework for Shared Memory[C]//Proceedings of the 18th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP '13), 2013.
[27]
Shakerinava M, Bakhshalipour M, Lotfi-kamran P, et al. Multi-lookahead offset prefetching[R], 2019.
[28]
Li S, Ahn J H, Strong R D, et al. McPAT: an integrated power, area, and timing modeling framework for multicore and manycore architectures[C]// 2009 42nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO),2009.
PDF(35936 KB)

Accesses

Citation

Detail

Sections
Recommended

/