• Home
• People
• Jobs
• Visit
• Thesis Topics
• Publications
• Conference Rankings
• Research
• Bcast
• Teaching
• Contact

SPONSORS

COLLABORATIONS

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

N. Dryden, R. Böhringer, T. Ben-Nun, T. Hoefler:
	Clairvoyant Prefetching for Distributed Machine Learning I/O (In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC21), presented in St. Louis, Missouri, ACM, Nov. 2021) Publisher Reference Abstract I/O is emerging as a major bottleneck for machine learning training, especially in distributed environments. Indeed, at large scale, I/O takes as much as 85% of training time. Addressing this I/O bottleneck necessitates careful optimization, as optimal data ingestion pipelines differ between systems, and require a delicate balance between access to local storage, external filesystems, and remote nodes. We introduce NoPFS, a machine learning I/O middleware, which provides a scalable, flexible, and easy-to-use solution to the I/O bottleneck. NoPFS uses clairvoyance: Given the seed generating the random access pattern for training with SGD, it can exactly predict when and where a sample will be accessed. We combine this with an analysis of access patterns and a performance model to provide distributed caching policies that adapt to different datasets and storage hierarchies. NoPFS reduces I/O times and improves end-to-end training by up to 5.4x on the ImageNet-1k, ImageNet-22k, and CosmoFlow datasets. Documents download article: access preprint on arxiv: download slides: Recorded talk (best effort)
	BibTeX
@inproceedings{nopfs,   author={Nikoli Dryden and Roman Böhringer and Tal Ben-Nun and Torsten Hoefler},   title={{Clairvoyant Prefetching for Distributed Machine Learning I/O}},   year={2021},   month={11},   booktitle={Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC21)},   location={St. Louis, Missouri},   publisher={ACM},   doi={https://doi.org/10.1145/3458817.3476181}, }