Express-Lane Scheduling and Multithreading to Minimize the Tail Latency of Microservices

Managing high-percentile tail latencies is key to designing user-facing cloud microservices. A main contributor to end-to-end tail latency is queuing, wherein nominal tasks are enqueued behind rare, long ones, due to head-of-line blocking. In this paper, we propose Express-Lane SMT (ESMT), which ext...

Full description

Saved in:
Bibliographic Details
Main Authors: Mirhosseini, Amirhossein, L. West, Brendan, W. Blake, Geoffrey, F. Wenisch, Thomas
Format: Conference Proceeding
Language:English
Subjects:
Cloud
Context
Hardware
Instruction sets
microservices
Multicore processing
Multithreading
OLDI
Organizations
scheduling
Servers
SMT
Task analysis
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by
cites
container_end_page 199
container_issue
container_start_page 194
container_title
container_volume
creator Mirhosseini, Amirhossein
L. West, Brendan
W. Blake, Geoffrey
F. Wenisch, Thomas
description Managing high-percentile tail latencies is key to designing user-facing cloud microservices. A main contributor to end-to-end tail latency is queuing, wherein nominal tasks are enqueued behind rare, long ones, due to head-of-line blocking. In this paper, we propose Express-Lane SMT (ESMT), which extends the hardware scheduling of a simultaneously multithreaded (SMT) core to provide an "express-lane" execution context for short tasks, protecting them from queuing behind rare, long ones. As tasks reach predefined service cutoffs, ESMT preempts and migrates them to the subsequent queue to be serviced by the next SMT execution lane, thereby preventing Head-of-Line (HoL) blocking. We further propose an enhanced variant of ESMT that allows execution lanes to work-steal from each other to maximize utilization. Our evaluation shows that ESMT with work stealing reduces tail latency over a conventional SMT core by an average of 56% and 67% under moderate (40%) and high (70%) system loads, respectively.
doi_str_mv 10.1109/ICAC.2019.00031
format conference_proceeding
fullrecord <record><control><sourceid>ieee_CHZPO</sourceid><recordid>TN_cdi_ieee_primary_8831216</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><ieee_id>8831216</ieee_id><sourcerecordid>8831216</sourcerecordid><originalsourceid>FETCH-LOGICAL-i175t-3e5da8301bfc9f7b41917fd51f53d1e3682c5d0802dc0d9235361ec0c44790c63</originalsourceid><addsrcrecordid>eNotjkFLwzAYhqMgOOfOHrzkD3R-X9I0yVHKdEKHByd4G1ny1Ua6djSdOH-9Ez298Dzw8DJ2gzBHBHv3VN6XcwFo5wAg8YzNrDaohUGRI-pzNhG5zjPQqrhkVyl9ACgLhZqwt8XXfqCUssp1xF98Q-HQxu6duy7w1aEd49gM5MIvGnu-il3cxW_iY0N87WLLKzdS54-8r0_SD32i4TN6StfsonZtotn_Ttnrw2JdLrPq-fF0t8oiajVmklRwRgJua29rvc3Roq6DwlrJgCQLI7wKYEAED8EKqWSB5MHnubbgCzllt3_dSESb_RB3bjhujJEosJA_smdRVQ</addsrcrecordid><sourcetype>Publisher</sourcetype><iscdi>true</iscdi><recordtype>conference_proceeding</recordtype></control><display><type>conference_proceeding</type><title>Express-Lane Scheduling and Multithreading to Minimize the Tail Latency of Microservices</title><source>IEEE Xplore All Conference Series</source><creator>Mirhosseini, Amirhossein ; L. West, Brendan ; W. Blake, Geoffrey ; F. Wenisch, Thomas</creator><creatorcontrib>Mirhosseini, Amirhossein ; L. West, Brendan ; W. Blake, Geoffrey ; F. Wenisch, Thomas</creatorcontrib><description>Managing high-percentile tail latencies is key to designing user-facing cloud microservices. A main contributor to end-to-end tail latency is queuing, wherein nominal tasks are enqueued behind rare, long ones, due to head-of-line blocking. In this paper, we propose Express-Lane SMT (ESMT), which extends the hardware scheduling of a simultaneously multithreaded (SMT) core to provide an "express-lane" execution context for short tasks, protecting them from queuing behind rare, long ones. As tasks reach predefined service cutoffs, ESMT preempts and migrates them to the subsequent queue to be serviced by the next SMT execution lane, thereby preventing Head-of-Line (HoL) blocking. We further propose an enhanced variant of ESMT that allows execution lanes to work-steal from each other to maximize utilization. Our evaluation shows that ESMT with work stealing reduces tail latency over a conventional SMT core by an average of 56% and 67% under moderate (40%) and high (70%) system loads, respectively.</description><identifier>EISSN: 2474-0756</identifier><identifier>EISBN: 9781728124117</identifier><identifier>EISBN: 1728124115</identifier><identifier>DOI: 10.1109/ICAC.2019.00031</identifier><language>eng</language><publisher>IEEE</publisher><subject>Cloud ; Context ; Hardware ; Instruction sets ; microservices ; Multicore processing ; Multithreading ; OLDI ; Organizations ; scheduling ; Servers ; SMT ; Task analysis</subject><ispartof>2019 IEEE International Conference on Autonomic Computing (ICAC), 2019, p.194-199</ispartof><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://ieeexplore.ieee.org/document/8831216$$EHTML$$P50$$Gieee$$H</linktohtml><link.rule.ids>309,310,780,784,789,790,27925,54555,54932</link.rule.ids><linktorsrc>$$Uhttps://ieeexplore.ieee.org/document/8831216$$EView_record_in_IEEE$$FView_record_in_$$GIEEE</linktorsrc></links><search><creatorcontrib>Mirhosseini, Amirhossein</creatorcontrib><creatorcontrib>L. West, Brendan</creatorcontrib><creatorcontrib>W. Blake, Geoffrey</creatorcontrib><creatorcontrib>F. Wenisch, Thomas</creatorcontrib><title>Express-Lane Scheduling and Multithreading to Minimize the Tail Latency of Microservices</title><title>2019 IEEE International Conference on Autonomic Computing (ICAC)</title><addtitle>ICAC</addtitle><description>Managing high-percentile tail latencies is key to designing user-facing cloud microservices. A main contributor to end-to-end tail latency is queuing, wherein nominal tasks are enqueued behind rare, long ones, due to head-of-line blocking. In this paper, we propose Express-Lane SMT (ESMT), which extends the hardware scheduling of a simultaneously multithreaded (SMT) core to provide an "express-lane" execution context for short tasks, protecting them from queuing behind rare, long ones. As tasks reach predefined service cutoffs, ESMT preempts and migrates them to the subsequent queue to be serviced by the next SMT execution lane, thereby preventing Head-of-Line (HoL) blocking. We further propose an enhanced variant of ESMT that allows execution lanes to work-steal from each other to maximize utilization. Our evaluation shows that ESMT with work stealing reduces tail latency over a conventional SMT core by an average of 56% and 67% under moderate (40%) and high (70%) system loads, respectively.</description><subject>Cloud</subject><subject>Context</subject><subject>Hardware</subject><subject>Instruction sets</subject><subject>microservices</subject><subject>Multicore processing</subject><subject>Multithreading</subject><subject>OLDI</subject><subject>Organizations</subject><subject>scheduling</subject><subject>Servers</subject><subject>SMT</subject><subject>Task analysis</subject><issn>2474-0756</issn><isbn>9781728124117</isbn><isbn>1728124115</isbn><fulltext>true</fulltext><rsrctype>conference_proceeding</rsrctype><creationdate>2019</creationdate><recordtype>conference_proceeding</recordtype><sourceid>6IE</sourceid><recordid>eNotjkFLwzAYhqMgOOfOHrzkD3R-X9I0yVHKdEKHByd4G1ny1Ua6djSdOH-9Ez298Dzw8DJ2gzBHBHv3VN6XcwFo5wAg8YzNrDaohUGRI-pzNhG5zjPQqrhkVyl9ACgLhZqwt8XXfqCUssp1xF98Q-HQxu6duy7w1aEd49gM5MIvGnu-il3cxW_iY0N87WLLKzdS54-8r0_SD32i4TN6StfsonZtotn_Ttnrw2JdLrPq-fF0t8oiajVmklRwRgJua29rvc3Roq6DwlrJgCQLI7wKYEAED8EKqWSB5MHnubbgCzllt3_dSESb_RB3bjhujJEosJA_smdRVQ</recordid><startdate>201906</startdate><enddate>201906</enddate><creator>Mirhosseini, Amirhossein</creator><creator>L. West, Brendan</creator><creator>W. Blake, Geoffrey</creator><creator>F. Wenisch, Thomas</creator><general>IEEE</general><scope>6IE</scope><scope>6IL</scope><scope>CBEJK</scope><scope>RIE</scope><scope>RIL</scope></search><sort><creationdate>201906</creationdate><title>Express-Lane Scheduling and Multithreading to Minimize the Tail Latency of Microservices</title><author>Mirhosseini, Amirhossein ; L. West, Brendan ; W. Blake, Geoffrey ; F. Wenisch, Thomas</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-i175t-3e5da8301bfc9f7b41917fd51f53d1e3682c5d0802dc0d9235361ec0c44790c63</frbrgroupid><rsrctype>conference_proceedings</rsrctype><prefilter>conference_proceedings</prefilter><language>eng</language><creationdate>2019</creationdate><topic>Cloud</topic><topic>Context</topic><topic>Hardware</topic><topic>Instruction sets</topic><topic>microservices</topic><topic>Multicore processing</topic><topic>Multithreading</topic><topic>OLDI</topic><topic>Organizations</topic><topic>scheduling</topic><topic>Servers</topic><topic>SMT</topic><topic>Task analysis</topic><toplevel>online_resources</toplevel><creatorcontrib>Mirhosseini, Amirhossein</creatorcontrib><creatorcontrib>L. West, Brendan</creatorcontrib><creatorcontrib>W. Blake, Geoffrey</creatorcontrib><creatorcontrib>F. Wenisch, Thomas</creatorcontrib><collection>IEEE Electronic Library (IEL) Conference Proceedings</collection><collection>IEEE Proceedings Order Plan All Online (POP All Online) 1998-present by volume</collection><collection>IEEE Xplore All Conference Proceedings</collection><collection>IEEE/IET Electronic Library</collection><collection>IEEE Proceedings Order Plans (POP All) 1998-Present</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext_linktorsrc</fulltext></delivery><addata><au>Mirhosseini, Amirhossein</au><au>L. West, Brendan</au><au>W. Blake, Geoffrey</au><au>F. Wenisch, Thomas</au><format>book</format><genre>proceeding</genre><ristype>CONF</ristype><atitle>Express-Lane Scheduling and Multithreading to Minimize the Tail Latency of Microservices</atitle><btitle>2019 IEEE International Conference on Autonomic Computing (ICAC)</btitle><stitle>ICAC</stitle><date>2019-06</date><risdate>2019</risdate><spage>194</spage><epage>199</epage><pages>194-199</pages><eissn>2474-0756</eissn><eisbn>9781728124117</eisbn><eisbn>1728124115</eisbn><abstract>Managing high-percentile tail latencies is key to designing user-facing cloud microservices. A main contributor to end-to-end tail latency is queuing, wherein nominal tasks are enqueued behind rare, long ones, due to head-of-line blocking. In this paper, we propose Express-Lane SMT (ESMT), which extends the hardware scheduling of a simultaneously multithreaded (SMT) core to provide an "express-lane" execution context for short tasks, protecting them from queuing behind rare, long ones. As tasks reach predefined service cutoffs, ESMT preempts and migrates them to the subsequent queue to be serviced by the next SMT execution lane, thereby preventing Head-of-Line (HoL) blocking. We further propose an enhanced variant of ESMT that allows execution lanes to work-steal from each other to maximize utilization. Our evaluation shows that ESMT with work stealing reduces tail latency over a conventional SMT core by an average of 56% and 67% under moderate (40%) and high (70%) system loads, respectively.</abstract><pub>IEEE</pub><doi>10.1109/ICAC.2019.00031</doi><tpages>6</tpages></addata></record>
fulltext fulltext_linktorsrc
identifier EISSN: 2474-0756
ispartof 2019 IEEE International Conference on Autonomic Computing (ICAC), 2019, p.194-199
issn 2474-0756
language eng
recordid cdi_ieee_primary_8831216
source IEEE Xplore All Conference Series
subjects Cloud
Context
Hardware
Instruction sets
microservices
Multicore processing
Multithreading
OLDI
Organizations
scheduling
Servers
SMT
Task analysis
title Express-Lane Scheduling and Multithreading to Minimize the Tail Latency of Microservices
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2025-01-08T04%3A18%3A25IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-ieee_CHZPO&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=proceeding&rft.atitle=Express-Lane%20Scheduling%20and%20Multithreading%20to%20Minimize%20the%20Tail%20Latency%20of%20Microservices&rft.btitle=2019%20IEEE%20International%20Conference%20on%20Autonomic%20Computing%20(ICAC)&rft.au=Mirhosseini,%20Amirhossein&rft.date=2019-06&rft.spage=194&rft.epage=199&rft.pages=194-199&rft.eissn=2474-0756&rft_id=info:doi/10.1109/ICAC.2019.00031&rft.eisbn=9781728124117&rft.eisbn_list=1728124115&rft_dat=%3Cieee_CHZPO%3E8831216%3C/ieee_CHZPO%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-i175t-3e5da8301bfc9f7b41917fd51f53d1e3682c5d0802dc0d9235361ec0c44790c63%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_id=info:pmid/&rft_ieee_id=8831216&rfr_iscdi=true