Copyright Notice:

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

J. Li, J. Dann, Z. He, G. Alonso, S. Rahul Chalamalasetti, D. Milojicic, L. Evans, A. Veprinsky, R. Shi:

 StreamDedup: Distributed In-line Deduplication for Disaggregated Storage

(ACM Trans. Reconfigurable Technol. Syst.. Vol , Nr. , In , presented in New York, NY, USA, pages , ACM, ISSN: 1936-7406, ISBN: , Mar. 2026, Just Accepted )

Publisher Reference

Abstract

Efficient data reduction techniques, including deduplication and compression, are essential in storage systems, affecting performance and longevity. Existing data deduplication approaches often focus on intra-SSD deduplication, missing opportunities for cross-node deduplication, or have scalability issues when aiming for low latency and high throughput data reduction on large-scale, distributed SSD arrays. We propose StreamDedup, a distributed stream accelerator implementing a transparent layer of deduplication as a network-attached, middle-tier service between the compute and storage tiers. StreamDedup manages all aspects of data deduplication and compression and can be seamlessly integrated into existing systems. It is RDMA-enabled and highly scalable, enhancing data processing capacities for large-scale storage systems. Our prototype, deployed on FPGAs, demonstrates that StreamDedup achieves a throughput of 12.7 GB/s on a single node, matching the network bandwidth of disaggregated storage, with a latency of less than 50 μs. Across 10 nodes StreamDedup shows an almost linear increase in throughput with less than 60 μs of latency.

Documents

download article:
download slides:
 

BibTeX

@article{li2026streamdedup,
  author={Jiayong Li and Jonas Dann and Zhenhao He and Gustavo Alonso and Sai Rahul Chalamalasetti and Dejan Milojicic and Lance Evans and Alex Veprinsky and Runbin Shi},
  title={{StreamDedup: Distributed In-line Deduplication for Disaggregated Storage}},
  journal={ACM Trans. Reconfigurable Technol. Syst.},
  institution={ETH Zurich},
  year={2026},
  month={03},
  pages={},
  volume={},
  number={},
  booktitle={},
  location={New York, NY, USA},
  publisher={ACM},
  issn={1936-7406},
  isbn={},
  note={Just Accepted},
  doi={0.1145/3799896},
}