From dev-return-100762-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Fri Jan 4 20:34:00 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id C0B97180660 for ; Fri, 4 Jan 2019 20:33:59 +0100 (CET) Received: (qmail 72963 invoked by uid 500); 4 Jan 2019 19:33:53 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 72951 invoked by uid 99); 4 Jan 2019 19:33:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jan 2019 19:33:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id AEEDAC9238 for ; Fri, 4 Jan 2019 19:33:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.049 X-Spam-Level: ** X-Spam-Status: No, score=2.049 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, KAM_LOTSOFHASH=0.25, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Sra7Jh_e-X-E for ; Fri, 4 Jan 2019 19:33:49 +0000 (UTC) Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 89F405F35B for ; Fri, 4 Jan 2019 19:33:49 +0000 (UTC) Received: by mail-ot1-f41.google.com with SMTP id 32so32947361ota.12 for ; Fri, 04 Jan 2019 11:33:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=x+hZCJK3H51HbO075gJ7bdSErzsEP78iQ3ASMy5dF5E=; b=PrbBJq6B01g+k4SHaWd2jrjl5SjOG8uyi5oDwflc8zLSWp39kzCPK3fMUmmmScwflb Vglff/ZtNxbaBn5/mQyIhLag8Q4hLJ1yNANx3IQ148q4xu9kVllJOA4mDu+tJHQ4qaRx IA0D5jUEEaueSVzL72t55hrYZkIjyw0thfv7UEElIaIP1CbxeLDF+J2hTdgkjKhWABnm KD2pypzXoEY9VSK4bCAy6p9AvY9DIC4+o7ONhmtAjOu8oJlj24SdZDlytqERiqP9sEQ6 vK0sGNN86CC7bYkREYZCCz/GOD2PcCargj4aT5fQdddcRmOLXnLL7cl6kMAtN7PLr3WM 1Qzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=x+hZCJK3H51HbO075gJ7bdSErzsEP78iQ3ASMy5dF5E=; b=eBLPs7brmReKqMgj9JEYEyFBpXwtdO8Zh70brD1am2vNfpTPBJt5v+A0YjtP8tXgpL 0WlSpenO68BL+llf2Oo0+SHw+tFzYQFrk1vU6a+M4lGahRBw0lA9Xl7EF8zqeIHgbrMB Qk3G7Wj+nl7kykl6YtVY3hjve3e7qIcveD6i8xGFhaWwcvSelfr8jwIwa6ZaLi2sLwUy 1VlG+Rr4V4C23oI7puxF1Ke66Xp7zftCyaUUKsuKX0nONr7XIHhZlosFWdxyzr7MUcfP EhZlTAZUEVWyU6/LG9/Dgq+c2rnV7tHGy6Js3HO+sZLPqrx8F/nX3VaauUFEBqDINSCt mrIA== X-Gm-Message-State: AJcUukfwu3EHgl3MN/MW3z6lL2ZvS8L+4soW7c8qNgXfrcQTF7zxa6rq zAX3FyiZtSeyGRxF9EcmTQpd3gYpkA7P4mktOhZROt3g X-Google-Smtp-Source: ALg8bN5I7hagxid3lD7WB47CqIeXA7a2omXF8SLDf/pvIyTipfeySEGiN/BLpMyJj19qI18Ru0F9dAAUC3aEOgi1F0A= X-Received: by 2002:a9d:1982:: with SMTP id k2mr34635563otk.197.1546630428606; Fri, 04 Jan 2019 11:33:48 -0800 (PST) MIME-Version: 1.0 References: <9855343E-3023-4A9E-B2A8-5854E34C0B09@storageos.com> <99CBA6C5-41FB-4CC1-9867-ED7F439D97F8@appnexus.com> In-Reply-To: From: Guozhang Wang Date: Fri, 4 Jan 2019 11:33:37 -0800 Message-ID: Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances To: dev Content-Type: multipart/alternative; boundary="0000000000004bc660057ea6f532" --0000000000004bc660057ea6f532 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Boyang, I've made another pass on the wiki page again. One minor comment is on the "Server Behavior Changes" section, we should have a paragraph on the logical changes on handling new versions of LeaveGroupRequest (e.g. how to handle dynamic member v.s. static member etc). Other than that, I do not have further comments. I think we can continue the voting process after that. Guozhang On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen wrote: > Thanks Jason for the comment! I answered it on the discuss thread. > > Folks, could we continue the vote for this KIP? This is a very critical > improvement for our streaming system > stability and we need to get things rolling right at the start of 2019. > > Thank you for your time! > Boyang > > ________________________________ > From: Jason Gustafson > Sent: Tuesday, December 18, 2018 7:40 AM > To: dev > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > reduce consumer rebalances > > Hi Boyang, > > Thanks, the KIP looks good. Just one comment. > > The new schema for the LeaveGroup request is slightly odd since it is > handling both the single consumer use case and the administrative use cas= e. > I wonder we could make it consistent from a batching perspective. > > In other words, instead of this: > LeaveGroupRequest =3D> GroupId MemberId [GroupInstanceId] > > Maybe we could do this: > LeaveGroupRequest =3D> GroupId [GroupInstanceId MemberId] > > For dynamic members, GroupInstanceId could be empty, which is consistent > with JoinGroup. What do you think? > > Also, just for clarification, what is the expected behavior if the curren= t > memberId of a static member is passed to LeaveGroup? Will the static memb= er > be removed? I know the consumer will not do this, but we'll still have to > handle the case on the broker. > > Best, > Jason > > > On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen wrote: > > > Thanks Stanislav=EF=BC=81 > > > > Get Outlook for iOS > > > > ________________________________ > > From: Stanislav Kozlovski > > Sent: Monday, December 10, 2018 11:28 PM > > To: dev@kafka.apache.org > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > reduce consumer rebalances > > > > This is great work, Boyang. Thank you very much. > > > > +1 (non-binding) > > > > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen wrote= : > > > > > Hey there, could I get more votes on this thread? > > > > > > Thanks for the vote from Mayuresh and Mike :) > > > > > > Best, > > > Boyang > > > ________________________________ > > > From: Mayuresh Gharat > > > Sent: Thursday, December 6, 2018 10:53 AM > > > To: dev@kafka.apache.org > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > > reduce consumer rebalances > > > > > > +1 (non-binding) > > > > > > Thanks, > > > > > > Mayuresh > > > > > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger < > > mike.freyberger@xandr.com> > > > wrote: > > > > > > > +1 (non binding) > > > > > > > > =EF=BB=BFOn 12/4/18, 9:43 AM, "Patrick Williams" < > > patrick.williams@storageos.com > > > > > > > > wrote: > > > > > > > > Pls take me off this VOTE list > > > > > > > > Best, > > > > > > > > Patrick Williams > > > > > > > > Sales Manager, UK & Ireland, Nordics & Israel > > > > StorageOS > > > > +44 (0)7549 676279 > > > > patrick.williams@storageos.com > > > > > > > > 20 Midtown > > > > 20 Proctor Street > > > > Holborn > > > > London WC1V 6NX > > > > > > > > Twitter: @patch37 > > > > LinkedIn: linkedin.com/in/patrickwilliams4 < > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Flinked= in.com%2Fin%2Fpatrickwilliams4&data=3D02%7C01%7C%7C9b12ec4ce9ae4454db8a= 08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C63680110125299409= 2&sdata=3DipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3D&reserv= ed=3D0 > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fslack= .storageos.com%2F&data=3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%= 7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata= =3DhxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3D&reserved=3D0 > > > > > > > > > > > > > > > > On 03/12/2018, 17:34, "Guozhang Wang" wrote: > > > > > > > > Hello Boyang, > > > > > > > > I've browsed through the new wiki and there are still a couple of > > > > minor > > > > things to notice: > > > > > > > > 1. RemoveMemberFromGroupOptions seems not defined anywhere. > > > > > > > > 2. LeaveGroupRequest added a list of group instance id, but still > > > > keep the > > > > member id as a singleton; is that intentional? I think to make > > > the > > > > protocol > > > > consistent both member id and instance ids could be plural. > > > > > > > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm > > > > wondering if we > > > > can defer adding this while just add the corresponding calls of > > > the > > > > LeaveGroupRequest inside Streams until we have used it in > > > > production and > > > > hence have a better understanding on how flexible or extensible > > > if > > > > we want > > > > to add any cmd tools. The rationale is that if we do not > > > > necessarily need > > > > it now, we can always add it later with a more think-through API > > > > design, > > > > but if we add the tool in a rush, we may need to extend or modify > > > > it soon > > > > after we realize its limits in operations. > > > > > > > > Otherwise, I'm +1 on the proposal. > > > > > > > > Guozhang > > > > > > > > > > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen > > > > wrote: > > > > > > > > > Hey community friends, > > > > > > > > > > after another month of polishing, KIP-345< > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bsta= tic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data= =3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaa= aaaaaaaa%7C1%7C0%7C636801101252994092&sdata=3DT4i7L1i0nIeHrrjTeLOOgYKsf= zfNEMGDhTazvBEZbXw%3D&reserved=3D0 > > > > > > > > > > design is ready for vote. Feel free to add your comment on the > > > > discussion > > > > > thread or here. > > > > > > > > > > Thanks for your time! > > > > > > > > > > Boyang > > > > > ________________________________ > > > > > From: Boyang Chen > > > > > Sent: Friday, November 9, 2018 6:35 AM > > > > > To: dev@kafka.apache.org > > > > > Subject: [VOTE] KIP-345: Introduce static membership protocol > > > to > > > > reduce > > > > > consumer rebalances > > > > > > > > > > Hey all, > > > > > > > > > > > > > > > thanks so much for all the inputs on KIP-345 so far. The > > > > original proposal > > > > > has enhanced a lot with your help. To make sure the > > > > implementation go > > > > > smoothly without back and forth, I would like to start a vote > > > on > > > > the final > > > > > design agreement now: > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=3D02%7C01%7C%7C9= b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%= 7C636801101252994092&sdata=3Dg4%2BMXKpkiQLZXg5HJWfJhw1kc1PbDNwyiX9zkREV= qGE%3D&reserved=3D0 > > > < > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bsta= tic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data= =3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaa= aaaaaaaa%7C1%7C0%7C636801101252994092&sdata=3DT4i7L1i0nIeHrrjTeLOOgYKsf= zfNEMGDhTazvBEZbXw%3D&reserved=3D0 > > > > > > > > > > > > > > > > > > > > > > > > > > 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances= < > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bsta= tic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data= =3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaa= aaaaaaaa%7C1%7C0%7C636801101252994092&sdata=3DT4i7L1i0nIeHrrjTeLOOgYKsf= zfNEMGDhTazvBEZbXw%3D&reserved=3D0 > > > > > > > > > > > > > > > > KIP-345: Introduce static membership protocol to reduce ...< > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bsta= tic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data= =3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaa= aaaaaaaa%7C1%7C0%7C636801101252994092&sdata=3DT4i7L1i0nIeHrrjTeLOOgYKsf= zfNEMGDhTazvBEZbXw%3D&reserved=3D0 > > > > > > > > > > > cwiki.apache.org > > > > > For stateful applications, one of the biggest performance > > > > bottleneck is > > > > > the state shuffling. In Kafka consumer, there is a concept > > > called > > > > > "rebalance" which means that for given M partitions and N > > > > consumers in one > > > > > consumer group, Kafka will try to balance the load between > > > > consumers and > > > > > ideally have ... > > > > > > > > > > > > > > > Let me know if you have any questions. > > > > > > > > > > > > > > > Best, > > > > > > > > > > Boyang > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > -Regards, > > > Mayuresh R. Gharat > > > (862) 250-7125 > > > > > > > > > -- > > Best, > > Stanislav > > > --=20 -- Guozhang --0000000000004bc660057ea6f532--