Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, M. Jaggi, D. Alistarh, T. Hoefler, J. Hensman:

 QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

(In Proceedings of the Neural Information Processing Systems, presented in Vancouver, Canada, Dec. 2024)

Abstract

We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4-bits, without any channels identified for retention in higher precision. Our quantized LLaMa2-70B model has losses of at most 0.29 WikiText-2 perplexity and retains 99% of the zero-shot performance.

Documents

access preprint on arxiv:
 

BibTeX

@inproceedings{QuaRot,
  author={Saleh Ashkboos and Amirkeivan Mohtashami and Maximilian L. Croci and Bo Li and Martin Jaggi and Dan Alistarh and Torsten Hoefler and James Hensman},
  title={{QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs}},
  year={2024},
  month={12},
  booktitle={Proceedings of the Neural Information Processing Systems},
  location={Vancouver, Canada},
}