Loading…
Pathways: Asynchronous Distributed Dataflow for ML
We present the design of a new large scale orchestration layer for accelerators. Our system, Pathways, is explicitly designed to enable exploration of new systems and ML research ideas, while retaining state of the art performance for current models. Pathways uses a sharded dataflow graph of asynchr...
Saved in:
Published in: | arXiv.org 2022-03 |
---|---|
Main Authors: | , , , , , , , , , , , , , , , |
Format: | Article |
Language: | English |
Subjects: | |
Online Access: | Get full text |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
cited_by | |
---|---|
cites | |
container_end_page | |
container_issue | |
container_start_page | |
container_title | arXiv.org |
container_volume | |
creator | Barham, Paul Chowdhery, Aakanksha Dean, Jeff Ghemawat, Sanjay Hand, Steven Hurt, Dan Isard, Michael Lim, Hyeontaek Pang, Ruoming Roy, Sudip Brennan Saeta Schuh, Parker Ryan Sepassi Laurent El Shafey Thekkath, Chandramohan A Wu, Yonghui |
description | We present the design of a new large scale orchestration layer for accelerators. Our system, Pathways, is explicitly designed to enable exploration of new systems and ML research ideas, while retaining state of the art performance for current models. Pathways uses a sharded dataflow graph of asynchronous operators that consume and produce futures, and efficiently gang-schedules heterogeneous parallel computations on thousands of accelerators while coordinating data transfers over their dedicated interconnects. Pathways makes use of a novel asynchronous distributed dataflow design that lets the control plane execute in parallel despite dependencies in the data plane. This design, with careful engineering, allows Pathways to adopt a single-controller model that makes it easier to express complex new parallelism patterns. We demonstrate that Pathways can achieve performance parity (~100% accelerator utilization) with state-of-the-art systems when running SPMD computations over 2048 TPUs, while also delivering throughput comparable to the SPMD case for Transformer models that are pipelined across 16 stages, or sharded across two islands of accelerators connected over a data center network. |
format | article |
fullrecord | <record><control><sourceid>proquest</sourceid><recordid>TN_cdi_proquest_journals_2642599443</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>2642599443</sourcerecordid><originalsourceid>FETCH-proquest_journals_26425994433</originalsourceid><addsrcrecordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mQwCkgsyShPrCy2UnAsrsxLzijKz8svLVZwySwuKcpMKi1JTVFwSSxJTMvJL1dIyy9S8PXhYWBNS8wpTuWF0twMym6uIc4eugVF-YWlqcUl8Vn5pUV5QKl4IzMTI1NLSxMTY2PiVAEA64Iz4w</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>2642599443</pqid></control><display><type>article</type><title>Pathways: Asynchronous Distributed Dataflow for ML</title><source>Publicly Available Content Database</source><creator>Barham, Paul ; Chowdhery, Aakanksha ; Dean, Jeff ; Ghemawat, Sanjay ; Hand, Steven ; Hurt, Dan ; Isard, Michael ; Lim, Hyeontaek ; Pang, Ruoming ; Roy, Sudip ; Brennan Saeta ; Schuh, Parker ; Ryan Sepassi ; Laurent El Shafey ; Thekkath, Chandramohan A ; Wu, Yonghui</creator><creatorcontrib>Barham, Paul ; Chowdhery, Aakanksha ; Dean, Jeff ; Ghemawat, Sanjay ; Hand, Steven ; Hurt, Dan ; Isard, Michael ; Lim, Hyeontaek ; Pang, Ruoming ; Roy, Sudip ; Brennan Saeta ; Schuh, Parker ; Ryan Sepassi ; Laurent El Shafey ; Thekkath, Chandramohan A ; Wu, Yonghui</creatorcontrib><description>We present the design of a new large scale orchestration layer for accelerators. Our system, Pathways, is explicitly designed to enable exploration of new systems and ML research ideas, while retaining state of the art performance for current models. Pathways uses a sharded dataflow graph of asynchronous operators that consume and produce futures, and efficiently gang-schedules heterogeneous parallel computations on thousands of accelerators while coordinating data transfers over their dedicated interconnects. Pathways makes use of a novel asynchronous distributed dataflow design that lets the control plane execute in parallel despite dependencies in the data plane. This design, with careful engineering, allows Pathways to adopt a single-controller model that makes it easier to express complex new parallelism patterns. We demonstrate that Pathways can achieve performance parity (~100% accelerator utilization) with state-of-the-art systems when running SPMD computations over 2048 TPUs, while also delivering throughput comparable to the SPMD case for Transformer models that are pipelined across 16 stages, or sharded across two islands of accelerators connected over a data center network.</description><identifier>EISSN: 2331-8422</identifier><language>eng</language><publisher>Ithaca: Cornell University Library, arXiv.org</publisher><subject>Accelerators ; Data centers</subject><ispartof>arXiv.org, 2022-03</ispartof><rights>2022. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.</rights><oa>free_for_read</oa><woscitedreferencessubscribed>false</woscitedreferencessubscribed></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktohtml>$$Uhttps://www.proquest.com/docview/2642599443?pq-origsite=primo$$EHTML$$P50$$Gproquest$$Hfree_for_read</linktohtml><link.rule.ids>780,784,25753,37012,44590</link.rule.ids></links><search><creatorcontrib>Barham, Paul</creatorcontrib><creatorcontrib>Chowdhery, Aakanksha</creatorcontrib><creatorcontrib>Dean, Jeff</creatorcontrib><creatorcontrib>Ghemawat, Sanjay</creatorcontrib><creatorcontrib>Hand, Steven</creatorcontrib><creatorcontrib>Hurt, Dan</creatorcontrib><creatorcontrib>Isard, Michael</creatorcontrib><creatorcontrib>Lim, Hyeontaek</creatorcontrib><creatorcontrib>Pang, Ruoming</creatorcontrib><creatorcontrib>Roy, Sudip</creatorcontrib><creatorcontrib>Brennan Saeta</creatorcontrib><creatorcontrib>Schuh, Parker</creatorcontrib><creatorcontrib>Ryan Sepassi</creatorcontrib><creatorcontrib>Laurent El Shafey</creatorcontrib><creatorcontrib>Thekkath, Chandramohan A</creatorcontrib><creatorcontrib>Wu, Yonghui</creatorcontrib><title>Pathways: Asynchronous Distributed Dataflow for ML</title><title>arXiv.org</title><description>We present the design of a new large scale orchestration layer for accelerators. Our system, Pathways, is explicitly designed to enable exploration of new systems and ML research ideas, while retaining state of the art performance for current models. Pathways uses a sharded dataflow graph of asynchronous operators that consume and produce futures, and efficiently gang-schedules heterogeneous parallel computations on thousands of accelerators while coordinating data transfers over their dedicated interconnects. Pathways makes use of a novel asynchronous distributed dataflow design that lets the control plane execute in parallel despite dependencies in the data plane. This design, with careful engineering, allows Pathways to adopt a single-controller model that makes it easier to express complex new parallelism patterns. We demonstrate that Pathways can achieve performance parity (~100% accelerator utilization) with state-of-the-art systems when running SPMD computations over 2048 TPUs, while also delivering throughput comparable to the SPMD case for Transformer models that are pipelined across 16 stages, or sharded across two islands of accelerators connected over a data center network.</description><subject>Accelerators</subject><subject>Data centers</subject><issn>2331-8422</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2022</creationdate><recordtype>article</recordtype><sourceid>PIMPY</sourceid><recordid>eNpjYuA0MjY21LUwMTLiYOAtLs4yMDAwMjM3MjU15mQwCkgsyShPrCy2UnAsrsxLzijKz8svLVZwySwuKcpMKi1JTVFwSSxJTMvJL1dIyy9S8PXhYWBNS8wpTuWF0twMym6uIc4eugVF-YWlqcUl8Vn5pUV5QKl4IzMTI1NLSxMTY2PiVAEA64Iz4w</recordid><startdate>20220323</startdate><enddate>20220323</enddate><creator>Barham, Paul</creator><creator>Chowdhery, Aakanksha</creator><creator>Dean, Jeff</creator><creator>Ghemawat, Sanjay</creator><creator>Hand, Steven</creator><creator>Hurt, Dan</creator><creator>Isard, Michael</creator><creator>Lim, Hyeontaek</creator><creator>Pang, Ruoming</creator><creator>Roy, Sudip</creator><creator>Brennan Saeta</creator><creator>Schuh, Parker</creator><creator>Ryan Sepassi</creator><creator>Laurent El Shafey</creator><creator>Thekkath, Chandramohan A</creator><creator>Wu, Yonghui</creator><general>Cornell University Library, arXiv.org</general><scope>8FE</scope><scope>8FG</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>AZQEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>DWQXO</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M7S</scope><scope>PIMPY</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PTHSS</scope></search><sort><creationdate>20220323</creationdate><title>Pathways: Asynchronous Distributed Dataflow for ML</title><author>Barham, Paul ; Chowdhery, Aakanksha ; Dean, Jeff ; Ghemawat, Sanjay ; Hand, Steven ; Hurt, Dan ; Isard, Michael ; Lim, Hyeontaek ; Pang, Ruoming ; Roy, Sudip ; Brennan Saeta ; Schuh, Parker ; Ryan Sepassi ; Laurent El Shafey ; Thekkath, Chandramohan A ; Wu, Yonghui</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-proquest_journals_26425994433</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2022</creationdate><topic>Accelerators</topic><topic>Data centers</topic><toplevel>online_resources</toplevel><creatorcontrib>Barham, Paul</creatorcontrib><creatorcontrib>Chowdhery, Aakanksha</creatorcontrib><creatorcontrib>Dean, Jeff</creatorcontrib><creatorcontrib>Ghemawat, Sanjay</creatorcontrib><creatorcontrib>Hand, Steven</creatorcontrib><creatorcontrib>Hurt, Dan</creatorcontrib><creatorcontrib>Isard, Michael</creatorcontrib><creatorcontrib>Lim, Hyeontaek</creatorcontrib><creatorcontrib>Pang, Ruoming</creatorcontrib><creatorcontrib>Roy, Sudip</creatorcontrib><creatorcontrib>Brennan Saeta</creatorcontrib><creatorcontrib>Schuh, Parker</creatorcontrib><creatorcontrib>Ryan Sepassi</creatorcontrib><creatorcontrib>Laurent El Shafey</creatorcontrib><creatorcontrib>Thekkath, Chandramohan A</creatorcontrib><creatorcontrib>Wu, Yonghui</creatorcontrib><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>Materials Science & Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>ProQuest Central Essentials</collection><collection>AUTh Library subscriptions: ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>ProQuest Central Korea</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Engineering Database</collection><collection>Publicly Available Content Database</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>Engineering Collection</collection></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Barham, Paul</au><au>Chowdhery, Aakanksha</au><au>Dean, Jeff</au><au>Ghemawat, Sanjay</au><au>Hand, Steven</au><au>Hurt, Dan</au><au>Isard, Michael</au><au>Lim, Hyeontaek</au><au>Pang, Ruoming</au><au>Roy, Sudip</au><au>Brennan Saeta</au><au>Schuh, Parker</au><au>Ryan Sepassi</au><au>Laurent El Shafey</au><au>Thekkath, Chandramohan A</au><au>Wu, Yonghui</au><format>book</format><genre>document</genre><ristype>GEN</ristype><atitle>Pathways: Asynchronous Distributed Dataflow for ML</atitle><jtitle>arXiv.org</jtitle><date>2022-03-23</date><risdate>2022</risdate><eissn>2331-8422</eissn><abstract>We present the design of a new large scale orchestration layer for accelerators. Our system, Pathways, is explicitly designed to enable exploration of new systems and ML research ideas, while retaining state of the art performance for current models. Pathways uses a sharded dataflow graph of asynchronous operators that consume and produce futures, and efficiently gang-schedules heterogeneous parallel computations on thousands of accelerators while coordinating data transfers over their dedicated interconnects. Pathways makes use of a novel asynchronous distributed dataflow design that lets the control plane execute in parallel despite dependencies in the data plane. This design, with careful engineering, allows Pathways to adopt a single-controller model that makes it easier to express complex new parallelism patterns. We demonstrate that Pathways can achieve performance parity (~100% accelerator utilization) with state-of-the-art systems when running SPMD computations over 2048 TPUs, while also delivering throughput comparable to the SPMD case for Transformer models that are pipelined across 16 stages, or sharded across two islands of accelerators connected over a data center network.</abstract><cop>Ithaca</cop><pub>Cornell University Library, arXiv.org</pub><oa>free_for_read</oa></addata></record> |
fulltext | fulltext |
identifier | EISSN: 2331-8422 |
ispartof | arXiv.org, 2022-03 |
issn | 2331-8422 |
language | eng |
recordid | cdi_proquest_journals_2642599443 |
source | Publicly Available Content Database |
subjects | Accelerators Data centers |
title | Pathways: Asynchronous Distributed Dataflow for ML |
url | http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-25T17%3A59%3A08IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest&rft_val_fmt=info:ofi/fmt:kev:mtx:book&rft.genre=document&rft.atitle=Pathways:%20Asynchronous%20Distributed%20Dataflow%20for%20ML&rft.jtitle=arXiv.org&rft.au=Barham,%20Paul&rft.date=2022-03-23&rft.eissn=2331-8422&rft_id=info:doi/&rft_dat=%3Cproquest%3E2642599443%3C/proquest%3E%3Cgrp_id%3Ecdi_FETCH-proquest_journals_26425994433%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=2642599443&rft_id=info:pmid/&rfr_iscdi=true |