Express-Lane Scheduling and Multithreading to Minimize the Tail Latency of Microservices

Managing high-percentile tail latencies is key to designing user-facing cloud microservices. A main contributor to end-to-end tail latency is queuing, wherein nominal tasks are enqueued behind rare, long ones, due to head-of-line blocking. In this paper, we propose Express-Lane SMT (ESMT), which ext...

Full description

Saved in:
Bibliographic Details
Main Authors: Mirhosseini, Amirhossein, L. West, Brendan, W. Blake, Geoffrey, F. Wenisch, Thomas
Format: Conference Proceeding
Language:English
Subjects:
Cloud
Context
Hardware
Instruction sets
microservices
Multicore processing
Multithreading
OLDI
Organizations
scheduling
Servers
SMT
Task analysis
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Managing high-percentile tail latencies is key to designing user-facing cloud microservices. A main contributor to end-to-end tail latency is queuing, wherein nominal tasks are enqueued behind rare, long ones, due to head-of-line blocking. In this paper, we propose Express-Lane SMT (ESMT), which extends the hardware scheduling of a simultaneously multithreaded (SMT) core to provide an "express-lane" execution context for short tasks, protecting them from queuing behind rare, long ones. As tasks reach predefined service cutoffs, ESMT preempts and migrates them to the subsequent queue to be serviced by the next SMT execution lane, thereby preventing Head-of-Line (HoL) blocking. We further propose an enhanced variant of ESMT that allows execution lanes to work-steal from each other to maximize utilization. Our evaluation shows that ESMT with work stealing reduces tail latency over a conventional SMT core by an average of 56% and 67% under moderate (40%) and high (70%) system loads, respectively.
ISSN:2474-0756
DOI:10.1109/ICAC.2019.00031