• Home
• People
• Jobs
• Visit
• Thesis Topics
• Publications
• Conference Rankings
• Research
• Bcast
• Teaching
• Contact

SPONSORS

COLLABORATIONS

The documents distributed by this server have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Publications of SPCL

K. Taranov, S. Byan, V. Marathe, T. Hoefler:
	KafkaDirect: Zero-copy Data Access for Apache Kafka over RDMA Networks (In Proceedings of the 2022 ACM SIGMOD International Conference on Management of Data, Jun. 2022) Publisher Reference Abstract Apache Kafka is an open-source distributed publish-subscribe system, which is widely used in data centers for messaging between applications, log aggregation, and stream processing. The existing Kafka implementation uses TCP/IP for communication, which has various inefficiencies such as a high message dispatch cost due to OS involvement and excessive memory copies. Recently, the availability of cost-effective RDMA-capable network controllers within data centers and cloud infrastructures have encouraged many modern applications to adopt RDMA networking, which offers the potential to outperform classical TCP/IP. We introduce KafkaDirect, an extension to Apache Kafka, that uses RDMA to accelerate the three most network intensive datapaths: record production, record replication, and record consumption. In this work, we explore the design choices including which RDMA operations to use to take full advantage of offloaded communication. Our RDMA design relies on one-sided RDMA requests to attain true zero-copy communication completely avoiding the need for using intermediate buffers in Kafka servers, thereby ensuring low latency and high throughput communication. KafkaDirect can offer up to 9x increase in throughput for both Kafka producers and Kafka consumers, and can provide 4x and 50x reduction in latency for Kafka producers and Kafka consumers, respectively. Documents download article: Recorded talk (best effort)
	BibTeX
@inproceedings{,   author={Konstantin Taranov and Steve Byan and Virendra Marathe and Torsten Hoefler},   title={{KafkaDirect: Zero-copy Data Access for Apache Kafka over RDMA Networks}},   year={2022},   month={06},   booktitle={Proceedings of the 2022 ACM SIGMOD International Conference on Management of Data},   doi={10.1145/3514221.3526056}, }