kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vahid S Hashemian" <vahidhashem...@us.ibm.com>
Subject Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets
Date Thu, 01 Feb 2018 20:40:44 GMT
Thanks James for sharing that scenario.

I agree it makes sense to be able to remove offsets for the topics that 
are no longer "active" in the group.
I think it becomes important to determine what constitutes that a topic is 
no longer active: If we use per-partition expiration we would manually 
choose a retention time that works for the particular scenario.

That works, but since we are manually intervening and specify a 
per-partition retention, why not do the intervention in some other way:

One alternative for this intervention, to favor the simplicity of the 
suggested protocol in the KIP, is to improve upon the just introduced 
DELETE_GROUPS API and allow for deletion of offsets of specific topics in 
the group. This is what the old ZooKeeper based group management supported 
anyway, and we would just be leveling the group deletion features of the 
Kafka-based group management with the ZooKeeper-based one.

So, instead of deciding in advance when the offsets should be removed we 
would instantly remove them when we are sure that they are no longer 

Let me know what you think.


From:   James Cheng <wushujames@gmail.com>
To:     dev@kafka.apache.org
Date:   02/01/2018 12:37 AM
Subject:        Re: [DISCUSS] KIP-211: Revise Expiration Semantics of 
Consumer Group Offsets


Under rejected alternatives, we had decided that we did NOT want to do 
per-partition expiration, and instead we wait until the entire group is 
empty and then (after the right time has passed) expire the entire group 
at once.

I thought of one scenario that might benefit from per-partition 

Let's say I have topics A B C... Z. So, I have 26 topics, all of them 
single partition, so 26 partitions. Let's say I have mirrormaker mirroring 
those 26 topics. The group will then have 26 committed offsets.

Let's say I then change the whitelist on mirrormaker so that it only 
mirrors topic Z, but I keep the same consumer group name. (I imagine that 
is a common thing to do?)

With the proposed design for this KIP, the committed offsets for topics A 
through Y will stay around as long as this mirroring group name exists.

In the current implementation that already exists (prior to this KIP), I 
belive that committed offsets for topics A through Y will expire.

How much do we care about this case?


> On Jan 23, 2018, at 11:44 PM, Jeff Widman <jeff@jeffwidman.com> wrote:
> Bumping this as I'd like to see it land...
> It's one of the "features" that tends to catch Kafka n00bs unawares and
> typically results in message skippage/loss, vs the proposed solution is
> much more intuitive behavior.
> Plus it's more wire efficient because consumers no longer need to commit
> offsets for partitions that have no new messages just to keep those 
> alive.
> On Fri, Jan 12, 2018 at 10:21 AM, Vahid S Hashemian <
> vahidhashemian@us.ibm.com> wrote:
>> There has been no further discussion on this KIP for about two months.
>> So I thought I'd provide the scoop hoping it would spark additional
>> feedback and move the KIP forward.
>> The KIP proposes a method to preserve group offsets as long as the 
>> is not in Empty state (even when offsets are committed very rarely), 
>> start the offset expiration of the group as soon as the group becomes
>> Empty.
>> It suggests dropping the `retention_time` field from the `OffsetCommit`
>> request and, instead, enforcing it via the broker config
>> `offsets.retention.minutes` for all groups. In other words, all groups
>> will have the same retention time.
>> The KIP presumes that this global retention config would suffice common
>> use cases and does not lead to, e.g., unmanageable offset cache size 
>> groups that don't need to stay around that long). It suggests opening
>> another KIP if this global retention setting proves to be problematic 
>> the future. It was suggested earlier in the discussion thread that the 
>> should propose a per-group retention config to circumvent this risk.
>> I look forward to hearing your thoughts. Thanks!
>> --Vahid
>> From:   "Vahid S Hashemian" <vahidhashemian@us.ibm.com>
>> To:     dev <dev@kafka.apache.org>
>> Date:   10/18/2017 04:45 PM
>> Subject:        [DISCUSS] KIP-211: Revise Expiration Semantics of 
>> Group Offsets
>> Hi all,
>> I created a KIP to address the group offset expiration issue reported 
>> KAFKA-4682:
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.
>> apache.org_confluence_display_KAFKA_KIP-2D211-253A-2BRevise-
>> 2BExpiration-2BSemantics-2Bof-2BConsumer-2BGroup-2BOffsets&
>> d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-
>> kjJc7uSVcviKUc&m=JkzH_2jfSMhCUPMk3rUasrjDAId6xbAEmX7_shSYdU4&s=
>> UBu7D2Obulg0fterYxL5m8xrDWkF_O2kGlygTCWsfFc&e=
>> Your feedback is welcome!
>> Thanks.
>> --Vahid
> -- 
> *Jeff Widman*
> jeffwidman.com <
> | 740-WIDMAN-J (943-6265)
> <><

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message