Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

A. Ivanov, N. Dryden, T. Hoefler:

 STen: An Interface for Efficient Sparsity in PyTorch

(In Sparsity in Neural Networks workshop, Jul. 2022)

Abstract

As deep learning models grow, sparsity is becoming an increasingly critical component of deep neural networks, enabling improved performance and reduced storage. However, existing frameworks offer poor support for sparsity. They primarily focus on sparse tensors in classical formats such as COO and CSR, which are not well suited to the sparsity regimes typical of deep learning, and neglect the broader sparsification pipeline necessary for using sparse models. To address this, we propose a new sparsity interface for PyTorch, STen, that incorporates sparsity layouts for tensors (including parameters and transients, e.g., activations), sparsity-aware operators, and sparsifiers, which define how a tensor is sparsified, and supports virtually all sparsification methods. STen can enable better sparse performance and simplify building sparse models, helping to make sparsity easily accessible.

Documents

download article:
download attachment:
 

BibTeX

@inproceedings{,
  author={Andrei Ivanov and Nikoli Dryden and Torsten Hoefler},
  title={{STen: An Interface for Efficient Sparsity in PyTorch}},
  year={2022},
  month={07},
  booktitle={Sparsity in Neural Networks workshop},
  doi={https://doi.org/10.48550/arXiv.2304.07613},
}