kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexei Zenin <alexei.ze...@mail.utoronto.ca>
Subject Possible Feature: Topic Retention Policy
Date Sat, 14 Oct 2017 14:48:48 GMT

I have come across a few stack overflow posts on the subject of request-response type semantics
through KAFKA. From some of the approaches that I've read developers are using KAFKA's auto
topic create feature (or AdminClient) to dynamically create topics per request-response channel.
They mention that the number of processes that wish to communicate can vary, with the processes
using some form of id to create unique topics for themselves. See https://stackoverflow.com/questions/35535785/does-kafka-support-request-response-messaging
and https://stackoverflow.com/questions/46662102/correlating-in-kafka-and-dynamic-topics/46678198<https://stackoverflow.com/questions/46662102/correlating-in-kafka-and-dynamic-topics/46678198#46678198>.

This approach leads to several problems however:

  1.  The maximum number of such channels (topics) is limited by the memory of a ZK node (in-memory
constraints since ZK is not sharded)
  2.  ZK is best used when reads outnumber writes. Creating topics at high rates could affect
ZK cluster performance.
  3.  Once the communication is done by the initiating process some entity must delete the
topic it used since it will never be used again

To solve one part of this problem, I find it strange that KAFKA does not provide a Topic Retention
Policy (not a log retention policy).

This would delete topics which are considered "stale" from the KAFKA cluster and from ZK.
By deleting topics for the user, this would reduce the amount of code and administrative headache
stale topics currently place on the user.

Would this be a feature that the community would find value in, while keeping true to KAFKA's
fundamentals and not require substantial refactoring?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message