kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vahid S Hashemian" <vahidhashem...@us.ibm.com>
Subject Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets
Date Tue, 03 Jul 2018 16:11:31 GMT
Hi Jason,

Thanks for the feedback. Your suggestions make sense to me. I think I'm 
more in favor of adding this info to kafka-topic tool (through another 
KIP) since it is not consumer group specific.
I'll wait for Gwen and others to comment before making changes to the KIP.

--Vahid




From:   Jason Gustafson <jason@confluent.io>
To:     dev <dev@kafka.apache.org>
Date:   06/28/2018 02:39 PM
Subject:        Re: [DISCUSS] KIP-325: Extend Consumer Group Command to 
Show Beginning Offsets



Hey Gwen/Vahid,

I think that use case makes sense, but isn't it a little odd to go through
the consumer group tool? I would expect to find something like that from
the kafka-topics tool instead. I don't feel too strongly about it, but I
hate to complicate the tool by adding the need to query topic configs. If
we don't have a meaningful number to report for compacted topics anyway,
then it feels like only a half solution. I'd probably suggest leaving this
out or just reporting the absolute difference even if a topic is 
compacted.

-Jason



On Thu, Jun 28, 2018 at 1:05 PM, Vahid S Hashemian <
vahidhashemian@us.ibm.com> wrote:

> Hi James,
>
>
>
> Thanks for the feedback. I updated the KIP and added some of the 
benefits
>
> of this improvement (including some that you mentioned).
>
>
>
> Regards.
>
> --Vahid
>
>
>
>
>
>
>
> From:   James Cheng <wushujames@gmail.com>
>
> To:     dev@kafka.apache.org
>
> Date:   06/27/2018 09:13 PM
>
> Subject:        Re: [DISCUSS] KIP-325: Extend Consumer Group Command to
>
> Show Beginning Offsets
>
>
>
>
>
>
>
> The “Motivation” section of the KIP says that the starting offset will 
be
>
> useful but doesn’t say why. Can you add a use-case or two to describe 
how
>
> it will be useful?
>
>
>
> In our case (see
>
> 
https://github.com/wushujames/kafka-utilities/blob/master/Co

> nsumerGroupLag/README.md
>
> ), we found the starting offset useful so that we could calculate
>
> partition size so that we could identify empty partitions (partitions
>
> where all the data had expired). In particular, we needed that info so
>
> that we could calculate “lag”. Consider that case where we are asked to
>
> mirror an abandoned topic where startOffset==endOffset==1000000. We 
would
>
> have no committed offset on it, and the topic has no data in it, so we
>
> won’t soon get any committed offset, and so “lag” is kind of undefined. 
We
>
> used the additional startOffset to allow us to detect this case.
>
>
>
> -James
>
>
>
> Sent from my iPhone
>
>
>
> > On Jun 26, 2018, at 11:23 AM, Vahid S Hashemian
>
> <vahidhashemian@us.ibm.com> wrote:
>
> >
>
> > Hi everyone,
>
> >
>
> > I have created a trivial KIP to improve the offset reporting of the
>
> > consumer group command:
>
> >
>
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%

> 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets
>
>
>
> > Looking forward to your feedback!
>
> >
>
> > Thanks.
>
> > --Vahid
>
> >
>
> >
>
>
>
>
>
>
>
>
>
>





Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message