metamodel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kaspersorensen <>
Subject [GitHub] metamodel issue #178: A new module for Apache Kafka "querying" using MetaMod...
Date Mon, 29 Jan 2018 03:36:55 GMT
Github user kaspersorensen commented on the issue:
    A bit of clarification to answer my own question/worry: It seems that the `Consumer.unsubscribe(..)`
call is the best thing I can reasonably do. This seems to remove any Zookeeper entry for the
consumer group. I think. At least on my own system where I'm testing, I can't see any trace
of my consumers in Zookeeper. But the things that I don't think we can ever get rid of are
the entries/records in the consumer_offsets topic that is internal to Kafka.
    I've been thinking that the best possibility would be to try and alleviate the problem
by reusing existing Consumer group IDs. Implemented by using some pool of group IDs where
leases for a group ID are granted when querying and returned when the DataSet is closed. But
then again, that would make concurrent queries (or even worse - queries from multiple processes
or nodes) very hard to manage because we would never know for sure that another MetaModel
client wasn't _also_ firing a query with the same Consumer group ID. So my conclusion so far
is that it isn't really improvable, except maybe by giving some hooks back to the user in
case he really worries about storing too much consumer group information.


View raw message