Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

P. Luczynski, L. Gianinazzi, P. Iff, L. Wilson, D. De Sensi, T. Hoefler:

 Near-Optimal Wafer-Scale Reduce

(In Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing (HPDC'24), presented in Pisa, Italy, Association for Computing Machinery, May 2024)

Abstract

Efficient Reduce and AllReduce communication collectives are a critical cornerstone of high-performance computing (HPC) applications. We present the first systematic investigation of Reduce and AllReduce on the Cerebras Wafer-Scale Engine (WSE). This architecture has been shown to achieve unprecedented performance both for machine learning workloads and other computational problems like FFT. We introduce a performance model to estimate the execution time of algorithms on the WSE and validate our predictions experimentally for a wide range of input sizes. In addition to existing implementations, we design and implement several new algorithms specifically tailored to the architecture. Moreover, we establish a lower bound for the runtime of a Reduce operation on the WSE. Based on our model, we automatically generate code that achieves near-optimal performance across the whole range of input sizes. Experiments demonstrate that our new Reduce and AllReduce algorithms outperform the current vendor solution by up to 3.27x. Additionally, our model predicts performance with less than 4% error. The proposed communication collectives increase the range of HPC applications that can benefit from the high throughput of the WSE. Our model-driven methodology demonstrates a disciplined approach that can lead the way to further algorithmic advancements on wafer-scale architectures.

Documents

download article:
access preprint on arxiv:
 

BibTeX

@inproceedings{,
  author={Piotr Luczynski and Lukas Gianinazzi and Patrick Iff and Leighton Wilson and Daniele De Sensi and Torsten Hoefler},
  title={{Near-Optimal Wafer-Scale Reduce}},
  year={2024},
  month={05},
  booktitle={Proceedings of the 33rd International Symposium on High-Performance Parallel and Distributed Computing (HPDC'24)},
  location={Pisa, Italy},
  publisher={Association for Computing Machinery},
}