Schedule

Introduction

Making systems fast and scalable: A compendium

Labor Day (No class)

Triton: An Intermediate Language and Compiler for Tiled Network Computation

Recommended background reading: Lam, Rothberg, Wolf, 1991, The cache performance and optimizations of blocked algorithms, https://doi.org/10.1145/106973.106981

An expanded version of the Triton paper is available as Tillet's PhD thesis, which is fairly readable (unlike most PhD theses).

Tillet, 2020, Blocked Algorithms for Neural Networks: Design and Implementation for GPUs, https://dash.harvard.edu/handle/1/37368966

Ansel, Jason, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, et al. 2024. ``PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation.'' In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, 929–47. La Jolla CA USA: ACM. https://doi.org/10.1145/3620665.3640366

Recommended background reading:

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, Ion Stoica, 2023, Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP '23: Proceedings of the 29th Symposium on Operating Systems Principles. https://doi.org/10.1145/3600006.3613165

Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu, Kai Li, and Song Han, 2024, DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), CVPR Open Access

Frantar, Elias, and Dan Alistarh. “QMoE: Sub-1-Bit Compression of Trillion-Parameter Models.” MLSys, 2024. PDF.

Cai, Zixian, Zhengyang Liu, Saeed Maleki, Madanlal Musuvathi, Todd Mytkowicz, Jacob Nelson, and Olli Saarikivi. “Synthesizing Optimal Collective Algorithms.” In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 62–75. Virtual Event Republic of Korea: ACM, 2021. https://doi.org/10.1145/3437801.3441620.

Related Reading

Weingram, Adam, Yuke Li, Hao Qi, Darren Ng, Liuyao Dai, and Xiaoyi Lu. “xCCL: A Survey of Industry-Led Collective Communication Libraries for Deep Learning.” Journal of Computer Science and Technology 38, no. 1 (February 2023): 166–95. https://doi.org/10.1007/s11390-023-2894-6.

Thakur, Rajeev, Rolf Rabenseifner, and William Gropp. “Optimization of Collective Communication Operations in MPICH.” The International Journal of High Performance Computing Applications 19, no. 1 (February 2005): 49–66. https://doi.org/10.1177/1094342005051521.

Jangda, Abhinav, Jun Huang, Guodong Liu, Amir Hossein Nodehi Sabet, Saeed Maleki, Youshan Miao, Madanlal Musuvathi, Todd Mytkowicz, and Olli Saarikivi. “Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads.” In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 402–16. Lausanne Switzerland: ACM, 2022. https://doi.org/10.1145/3503222.3507778.

Jouppi, Norm, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, et al. 2023. “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings.” In Proceedings of the 50th Annual International Symposium on Computer Architecture, 1–14. ISCA ’23. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3579371.3589350.

Background reading

Jouppi, Norman P., Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, et al. 2017. “In-Datacenter Performance Analysis of a Tensor Processing Unit.” In Proceedings of the 44th Annual International Symposium on Computer Architecture, 1–12. ISCA ’17. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3079856.3080246. (the original TPU paper)

This CACM article should be more accessible (highly recommended!): A Domain-Specific Supercomputer for Training Deep Neural Networks

Likewise, this IEEE Micro article: The Design Process for Google's Training Chips: TPUv2 and TPUv3

Abts, Dennis, Jonathan Ross, Jonathan Sparling, Mark Wong-VanHaren, Max Baker, Tom Hawkins, Andrew Bell, et al. 2020. “Think Fast: A Tensor Streaming Processor (TSP) for Accelerating Deep Learning Workloads.” In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 145–58. Valencia, Spain: IEEE. https://doi.org/10.1109/ISCA45697.2020.00023.

Summary Discussion

Midterm (in class)

Fall Break (No class)

Overview of Projects

ML Training Basics

Xu, Lijie, Shuang Qiu, Binhang Yuan, Jiawei Jiang, Cedric Renggli, Shaoduo Gan, Kaan Kara, et al. 2022. “In-Database Machine Learning with CorgiPile: Stochastic Gradient Descent without Full Data Shuffle.” In Proceedings of the 2022 International Conference on Management of Data, 1286–1300. Philadelphia PA USA: ACM. https://doi.org/10.1145/3514221.3526150.

Zhao, Mark, Satadru Pan, Niket Agarwal, Zhaoduo Wen, David Xu, Anand Natarajan, Pavan Kumar, et al. 2023. “{Tectonic-Shift}: A Composite Storage Fabric for {Large-Scale} {ML} Training.” In , 433–49. https://www.usenix.org/conference/atc23/presentation/zhao.

López-Paradís, Guillem, Isaac M. Hair, Sid Kannan, Roman Rabbat, Parker Murray, Alex Lopes, Rory Zahedi, Winston Zuo, and Jonathan Balkind. 2024. “The Case For Data Centre Hyperloops.” In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), 230–44. https://doi.org/10.1109/ISCA59077.2024.00026.

Choudhury, Arnab, Yang Wang, Tuomas Pelkonen, Kutta Srinivasan, Abha Jain, Shenghao Lin, Delia David, et al. 2024. “{MAST}: Global Scheduling of {ML} Training across {Geo-Distributed} Datacenters at Hyperscale.” In , 563–80. https://www.usenix.org/conference/osdi24/presentation/choudhury.

Shukla, Dharma, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen, et al. 2022. “Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads.” arXiv. https://doi.org/10.48550/arXiv.2202.07848

Zhao, Yanli, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, et al. 2023. “PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.” Proceedings of the VLDB Endowment 16 (12): 3848–60. https://doi.org/10.14778/3611540.3611569.

Li, Shen, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, et al. 2020. “PyTorch Distributed: Experiences on Accelerating Data Parallel Training.” Proceedings of the VLDB Endowment 13 (12): 3005–18. https://doi.org/10.14778/3415478.3415530.

Mid-project presentations (Prof. Pai travelling)

Guest lecture by Dr. Ryan Yates, Groq Inc. (In WH-2506)

Tang, Yuxin, Zhimin Ding, Dimitrije Jankov, Binhang Yuan, Daniel Bourgeois, and Chris Jermaine. “Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning.” In Proceedings of the 40th International Conference on Machine Learning, 33581–98. PMLR, 2023. https://proceedings.mlr.press/v202/tang23a.html.

Background Reading

Merriënboer, Bart van, Olivier Breuleux, Arnaud Bergeron, and Pascal Lamblin. “Automatic Differentiation in ML: Where We Are and Where We Should Be Going,” NeurIPS, 2018. https://proceedings.neurips.cc/paper_files/paper/2018/file/770f8e448d07586afbf77bb59f698587-Paper.pdf.

Discussion with Zeliang Zhang, URCS AI PhD student

Thanksgiving holiday starts, no class

Last day of class

Reading Period

Reading Period