Memory‐aware kernel mechanism and policies for improving internode load balancing on NUMA systems

• Published in: Software, practice & experience practice & experience, 2019-10, Vol.49 (10), p.1485-1508
• Main Authors: Chiang, Mei‐Ling, Su, Wei‐Lun, Tu, Shu‐Wei, Lin, Zhen‐Wei
• Format: Article
• Language: English
• Subjects: Central processing units; Computer memory; Connectivity; CPUs; Kernels; Linux; Load balancing; Mapping; Nodes; NUMA; Performance degradation; Performance enhancement; Policies; remote memory access; resource contention; scheduling; Servers
• ISSN: 0038-0644; 1097-024X
• DOI: 10.1002/spe.2731

Summary Although nonuniform memory access architecture provides better scalability for multicore systems, cores accessing memory on remote nodes take longer than those accessing on local nodes. Remote memory access accompanied by contention for internode interconnection degrades performance. Properly mapping threads to cores and data accessed to their nodes can substantially improve performance and energy efficiency. However, an operating system kernel's load‐balancing activity may migrate threads across nodes, which thus messes up the thread mapping. Besides, subsequent data mapping behavior pays for the cost of page migration to reduce remote memory access. Once unsuitable threads are migrated, it is detrimental to system performance. This paper focuses on improving the kernel's internode load balancing on nonuniform memory access systems. We develop a memory‐aware kernel mechanism and policies to reduce remote memory access incurred by internode thread migration. The Linux kernel's load balancing mechanism is modified to incorporate selection policies in the internode thread migration, and the kernel is modified to track the amount of memory used by each thread on each node. With this information, well‐designed policies can then choose suitable threads for internode migration. The purpose is to avoid migrating a thread that might incur relatively more remote memory access and page migration. The experimental results show that with our mechanism and the proposed selection policies, the system performance is substantially increased when compared with the unmodified Linux kernel that does not consider memory usage and always migrates the first‐fit thread in the runqueue that can be migrated to the target central processing unit.