Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

W. Jiang, M. Zeller, R. Waleffe, T. Hoefler, G. Alonso:

 Chameleon: A Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models

(Proc. VLDB Endow.. Vol 18, Nr. 1, ISSN: 2150-8097, Sep. 2024)

Publisher Reference

Abstract

A Retrieval-Augmented Language Model (RALM) combines a large language model (LLM) with a vector database to retrieve context-specific knowledge during text generation. This strategy facilitates impressive generation quality even with smaller models, thus reducing computational demands by orders of magnitude. To serve RALMs efficiently and flexibly, we propose Chameleon, a heterogeneous accelerator system integrating both LLM and vector search accelerators in a disaggregated architecture. The heterogeneity ensures efficient serving for both inference and retrieval, while the disaggregation allows independent scaling of LLM and vector search accelerators to fulfill diverse RALM requirements. Our Chameleon prototype implements vector search accelerators on FPGAs and assigns LLM inference to GPUs, with CPUs as cluster coordinators. Evaluated on various RALMs, Chameleon exhibits up to 2.16x reduction in latency and 3.18x speedup in throughput compared to the hybrid CPU-GPU architecture. The promising results pave the way for adopting heterogeneous accelerators for not only LLM inference but also vector search in future RALM systems.

Documents

download article:
access preprint on arxiv:
 

BibTeX

@article{,
  author={Wenqi Jiang and Marco Zeller and Roger Waleffe and Torsten Hoefler and Gustavo Alonso},
  title={{Chameleon: A Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models}},
  journal={Proc. VLDB Endow.},
  year={2024},
  month={9},
  volume={18},
  number={1},
  issn={2150-8097},
  doi={10.14778/3696435.3696439},
}