From dev-return-97175-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Wed Aug 15 18:51:00 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 909A8180626 for ; Wed, 15 Aug 2018 18:50:59 +0200 (CEST) Received: (qmail 53312 invoked by uid 500); 15 Aug 2018 16:50:58 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 53289 invoked by uid 99); 15 Aug 2018 16:50:57 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Aug 2018 16:50:57 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 4E80C18071F for ; Wed, 15 Aug 2018 16:50:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.499 X-Spam-Level: ** X-Spam-Status: No, score=2.499 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LINEPADDING=1.2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 3FgfVqpsN3FT for ; Wed, 15 Aug 2018 16:50:54 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 852815F3BE for ; Wed, 15 Aug 2018 16:50:53 +0000 (UTC) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w7FGd8Eh000545 for ; Wed, 15 Aug 2018 12:50:51 -0400 Received: from smtp.notes.na.collabserv.com (smtp.notes.na.collabserv.com [192.155.248.75]) by mx0a-001b2d01.pphosted.com with ESMTP id 2kvn0s0f2r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 15 Aug 2018 12:50:51 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Wed, 15 Aug 2018 16:50:49 -0000 Received: from us1a3-smtp06.a3.dal06.isc4sb.com (10.146.103.243) by smtp.notes.na.collabserv.com (10.106.227.123) with smtp.notes.na.collabserv.com ESMTP; Wed, 15 Aug 2018 16:50:46 -0000 Received: from us1a3-mail107.a3.dal06.isc4sb.com ([10.146.45.243]) by us1a3-smtp06.a3.dal06.isc4sb.com with ESMTP id 2018081516504556-774679 ; Wed, 15 Aug 2018 16:50:45 +0000 In-Reply-To: To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets From: "Vahid S Hashemian" Date: Wed, 15 Aug 2018 09:50:44 -0700 References: <55EDE47B-713E-4FA5-B6FB-A8987D5571EA@gmail.com> X-KeepSent: FD0296CA:388A5BE8-002582EA:005C244C; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1EXT SHF888 April 26, 2018 X-LLNOutbound: False X-Disclaimed: 1871 X-TNEFEvaluated: 1 x-cbid: 18081516-3815-0000-0000-000007B7F8D4 X-IBM-SpamModules-Scores: BY=0; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.433748; ST=0; TS=0; UL=0; ISC=; MB=0.025143 X-IBM-SpamModules-Versions: BY=3.00009550; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000266; SDB=6.01073944; UDB=6.00553399; IPR=6.00853902; BA=6.00006076; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00022736; XFM=3.00000015; UTC=2018-08-15 16:50:47 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2018-08-15 12:09:48 - 6.00008835 x-cbparentid: 18081516-3816-0000-0000-0000BF4F1410 Message-Id: Content-Type: multipart/alternative; boundary="=_alternative 005C8816882582EA_=" MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-08-15_06:,, signatures=0 X-Proofpoint-Spam-Reason: safe --=_alternative 005C8816882582EA_= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: Quoted-printable In the absence of additional feedback to upvote one option against the =0D other, I decided to roll the KIP back to an earlier version without the =0D "partition size" support.=0D I'll start another KIP to add the partition size info to kafka-topic =0D command output, where I think would be a better fit.=0D =0D If there is no additional comment/feedback I'll start a vote soon.=0D =0D Thanks.=0D --Vahid=0D =0D =0D =0D =0D From: "Vahid S Hashemian" =0D To: dev@kafka.apache.org=0D Date: 08/07/2018 01:58 PM=0D Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group Command to =0D Show Beginning Offsets=0D =0D =0D =0D Any additional feedback on whether we should also include a partition size = =0D =0D column or not?=0D =0D Options:=0D =0D 1. The current KIP (with a partition size column): =0D https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%3A+Extend+Consume= r+Group+Command+to+Show+Beginning+Offsets+and+Partition+Size=0D =0D ** Requires additional processing to extract topic configs=0D ** Will not return a precise size for compacted topics=0D ** Simplifies checking the consumption progress on each partition=0D =0D 2. The earlier version (without a partition size column): =0D https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=3D87296412= =0D =0D ** Less processing overhead compared to Option 1=0D ** Partition size can be provided via kafka-topics tool=0D =0D For further info, please see the recent discussions on the thread.=0D =0D Thanks!=0D --Vahid=0D =0D =0D =0D =0D From: "Vahid S Hashemian" =0D To: dev@kafka.apache.org=0D Date: 07/03/2018 09:11 AM=0D Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group Command to =0D Show Beginning Offsets=0D =0D =0D =0D Hi Jason,=0D =0D Thanks for the feedback. Your suggestions make sense to me. I think I'm =0D more in favor of adding this info to kafka-topic tool (through another =0D KIP) since it is not consumer group specific.=0D I'll wait for Gwen and others to comment before making changes to the KIP.= =0D =0D --Vahid=0D =0D =0D =0D =0D From: Jason Gustafson =0D To: dev =0D Date: 06/28/2018 02:39 PM=0D Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group Command to =0D Show Beginning Offsets=0D =0D =0D =0D Hey Gwen/Vahid,=0D =0D I think that use case makes sense, but isn't it a little odd to go through= =0D the consumer group tool? I would expect to find something like that from=0D the kafka-topics tool instead. I don't feel too strongly about it, but I=0D hate to complicate the tool by adding the need to query topic configs. If=0D we don't have a meaningful number to report for compacted topics anyway,=0D then it feels like only a half solution. I'd probably suggest leaving this= =0D out or just reporting the absolute difference even if a topic is =0D compacted.=0D =0D -Jason=0D =0D =0D =0D On Thu, Jun 28, 2018 at 1:05 PM, Vahid S Hashemian <=0D vahidhashemian@us.ibm.com> wrote:=0D =0D > Hi James,=0D >=0D >=0D >=0D > Thanks for the feedback. I updated the KIP and added some of the =0D benefits=0D >=0D > of this improvement (including some that you mentioned).=0D >=0D >=0D >=0D > Regards.=0D >=0D > --Vahid=0D >=0D >=0D >=0D >=0D >=0D >=0D >=0D > From: James Cheng =0D >=0D > To: dev@kafka.apache.org=0D >=0D > Date: 06/27/2018 09:13 PM=0D >=0D > Subject: Re: [DISCUSS] KIP-325: Extend Consumer Group Command to=0D >=0D > Show Beginning Offsets=0D >=0D >=0D >=0D >=0D >=0D >=0D >=0D > The =E2=80=9CMotivation=E2=80=9D section of the KIP says that the startin= g offset will =0D be=0D >=0D > useful but doesn=E2=80=99t say why. Can you add a use-case or two to desc= ribe =0D how=0D >=0D > it will be useful?=0D >=0D >=0D >=0D > In our case (see=0D >=0D > =0D https://github.com/wushujames/kafka-utilities/blob/master/Co=0D =0D =0D =0D > nsumerGroupLag/README.md=0D >=0D > ), we found the starting offset useful so that we could calculate=0D >=0D > partition size so that we could identify empty partitions (partitions=0D >=0D > where all the data had expired). In particular, we needed that info so=0D >=0D > that we could calculate =E2=80=9Clag=E2=80=9D. Consider that case where w= e are asked to=0D >=0D > mirror an abandoned topic where startOffset=3D=3DendOffset=3D=3D1000000. = We =0D would=0D >=0D > have no committed offset on it, and the topic has no data in it, so we=0D >=0D > won=E2=80=99t soon get any committed offset, and so =E2=80=9Clag=E2=80=9D= is kind of undefined. =0D We=0D >=0D > used the additional startOffset to allow us to detect this case.=0D >=0D >=0D >=0D > -James=0D >=0D >=0D >=0D > Sent from my iPhone=0D >=0D >=0D >=0D > > On Jun 26, 2018, at 11:23 AM, Vahid S Hashemian=0D >=0D > wrote:=0D >=0D > >=0D >=0D > > Hi everyone,=0D >=0D > >=0D >=0D > > I have created a trivial KIP to improve the offset reporting of the=0D >=0D > > consumer group command:=0D >=0D > >=0D >=0D > =0D https://cwiki.apache.org/confluence/display/KAFKA/KIP-325%=0D =0D =0D =0D > 3A+Extend+Consumer+Group+Command+to+Show+Beginning+Offsets=0D >=0D >=0D >=0D > > Looking forward to your feedback!=0D >=0D > >=0D >=0D > > Thanks.=0D >=0D > > --Vahid=0D >=0D > >=0D >=0D > >=0D >=0D >=0D >=0D >=0D >=0D >=0D >=0D >=0D >=0D >=0D =0D =0D =0D =0D =0D =0D =0D =0D =0D =0D =0D =0D --=_alternative 005C8816882582EA_=--