From dev-return-100770-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Sat Jan 5 00:50:47 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 071B4180660 for ; Sat, 5 Jan 2019 00:50:46 +0100 (CET) Received: (qmail 8965 invoked by uid 500); 4 Jan 2019 23:50:45 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 8950 invoked by uid 99); 4 Jan 2019 23:50:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Jan 2019 23:50:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 9635AC9E7E for ; Fri, 4 Jan 2019 23:50:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.05 X-Spam-Level: ** X-Spam-Status: No, score=2.05 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, KAM_LOTSOFHASH=0.25, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=confluent.io Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id d-4vB6_gurGd for ; Fri, 4 Jan 2019 23:50:41 +0000 (UTC) Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 6C89E5FAEB for ; Fri, 4 Jan 2019 23:50:41 +0000 (UTC) Received: by mail-ot1-f52.google.com with SMTP id 40so33448635oth.4 for ; Fri, 04 Jan 2019 15:50:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=confluent.io; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=fw9lHdQIGfK83fw3aC5XPAxyJwRDvWtxMAg3AGyA+p4=; b=VKg6tiW0YJWQ1BTTUPSKszmvOVN3ula8X1LSbFCQXfg4D8b5AF3y6z+xAYovk0YBMf b+DQhIe/Y85Eu/SVK+AtgwgQ7M4DdwbsyV2GbogZ7l0lQ5B80W1LV/72RlrWSfWjdrjB OcAtFEOO+BL+kSsQ2a/TiuK5lV9N8QIiwF6kvozRz6Ifna9jjAWf0krAUUt7+vXKVZbE IiwjnJb/mAVnUIcy4VHN8lgZkZLr592RhRUCRm4Gpw+yPRXzu9C1qk5o4FFBrRjQ+tQ4 Jqs63zewh8Jjzb6dTnOXqOcabKHcY1qsIAkvxLAqgptLgJFjMI+HtOiAlPOuqEXV6bC5 4+4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=fw9lHdQIGfK83fw3aC5XPAxyJwRDvWtxMAg3AGyA+p4=; b=EuwxM8KEPzkmFdsoQCD64Cba2gNzodMD1hw4bTnhqDChk25LGGvI/iNKuLv+6wKTYA JrrT86v5Gt4DBrNWDD7NvFSV/vkp0UY6sd/zYg4znpUhVlSWSpqKOEWf1Hi2hYGM9fZp 7RY3NRzoG99EPqJ21Pl0i2FWBRNsqIaEo6AVNlZAeeO7OuBbhMFQ4kIRp82GGnlDW1Sn zJ8g/Y3bS6ASf+KSHh/6981/FBwUum+JAXMsTGlp5ZuGNyc84ha4QOG2lpHqo8UJfpzv wdpX/bNpIULnoGxtH6YUSP8ikukiaMvbUQjxBh9UzYZk8BxPQ7PLdMqlmRxZIPG6YO/6 oLGQ== X-Gm-Message-State: AJcUukdc0tI8oTUzxzAqimGU8W/d7s44/CNYHdTlZ+m12gbk/GHthKBD MjJ4lYcbRuFN7R/fVvoZklJTBq7Dd6HbLmma4A6nVWTZCW8= X-Google-Smtp-Source: ALg8bN6NGU+d/QjrNvysuK9vRYFp1UQixV36e5VjldEMfi2cbilrnLAPeNdavb6KTQQf/4Ywp5pCbjxW/8NbvZJGMO0= X-Received: by 2002:a9d:4f0c:: with SMTP id d12mr38079798otl.302.1546645835139; Fri, 04 Jan 2019 15:50:35 -0800 (PST) MIME-Version: 1.0 References: <9855343E-3023-4A9E-B2A8-5854E34C0B09@storageos.com> <99CBA6C5-41FB-4CC1-9867-ED7F439D97F8@appnexus.com> In-Reply-To: From: Jason Gustafson Date: Fri, 4 Jan 2019 15:50:23 -0800 Message-ID: Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances To: dev Content-Type: multipart/alternative; boundary="00000000000098eb47057eaa8bda" --00000000000098eb47057eaa8bda Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hey Boyang, I had just a follow-up to my comment above. I wanted to suggest an alternative schema for LeaveGroup: LeaveGroupRequest =3D> GroupId [GroupInstanceId MemberId] So we have a single array instead of two arrays. Each element identifies a single member. For dynamic members, we would use GroupInstanceId=3D"" and provide the dynamic MemberId, which is consistent with JoinGroup. For static members, GroupInstanceId must be provided and Member could be considered optional. I think this makes the schema more coherent, but I'll leave it to you if there is a good reason to keep them separate. In any case, my vote is +1. Thanks for the hard work on this KIP! Best, Jason On Fri, Jan 4, 2019 at 1:09 PM Boyang Chen wrote: > Thanks Guozhang for the proposal! The update is done. > > ________________________________ > From: Guozhang Wang > Sent: Saturday, January 5, 2019 3:33 AM > To: dev > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > reduce consumer rebalances > > Hello Boyang, > > I've made another pass on the wiki page again. One minor comment is on th= e > "Server Behavior Changes" section, we should have a paragraph on the > logical changes on handling new versions of LeaveGroupRequest (e.g. how t= o > handle dynamic member v.s. static member etc). > > Other than that, I do not have further comments. I think we can continue > the voting process after that. > > Guozhang > > On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen wrote: > > > Thanks Jason for the comment! I answered it on the discuss thread. > > > > Folks, could we continue the vote for this KIP? This is a very critical > > improvement for our streaming system > > stability and we need to get things rolling right at the start of 2019. > > > > Thank you for your time! > > Boyang > > > > ________________________________ > > From: Jason Gustafson > > Sent: Tuesday, December 18, 2018 7:40 AM > > To: dev > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > reduce consumer rebalances > > > > Hi Boyang, > > > > Thanks, the KIP looks good. Just one comment. > > > > The new schema for the LeaveGroup request is slightly odd since it is > > handling both the single consumer use case and the administrative use > case. > > I wonder we could make it consistent from a batching perspective. > > > > In other words, instead of this: > > LeaveGroupRequest =3D> GroupId MemberId [GroupInstanceId] > > > > Maybe we could do this: > > LeaveGroupRequest =3D> GroupId [GroupInstanceId MemberId] > > > > For dynamic members, GroupInstanceId could be empty, which is consisten= t > > with JoinGroup. What do you think? > > > > Also, just for clarification, what is the expected behavior if the > current > > memberId of a static member is passed to LeaveGroup? Will the static > member > > be removed? I know the consumer will not do this, but we'll still have = to > > handle the case on the broker. > > > > Best, > > Jason > > > > > > On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen > wrote: > > > > > Thanks Stanislav=EF=BC=81 > > > > > > Get Outlook for iOS > > > > > > ________________________________ > > > From: Stanislav Kozlovski > > > Sent: Monday, December 10, 2018 11:28 PM > > > To: dev@kafka.apache.org > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > > reduce consumer rebalances > > > > > > This is great work, Boyang. Thank you very much. > > > > > > +1 (non-binding) > > > > > > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen > wrote: > > > > > > > Hey there, could I get more votes on this thread? > > > > > > > > Thanks for the vote from Mayuresh and Mike :) > > > > > > > > Best, > > > > Boyang > > > > ________________________________ > > > > From: Mayuresh Gharat > > > > Sent: Thursday, December 6, 2018 10:53 AM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol t= o > > > > reduce consumer rebalances > > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > > > > > Mayuresh > > > > > > > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger < > > > mike.freyberger@xandr.com> > > > > wrote: > > > > > > > > > +1 (non binding) > > > > > > > > > > =EF=BB=BFOn 12/4/18, 9:43 AM, "Patrick Williams" < > > > patrick.williams@storageos.com > > > > > > > > > > wrote: > > > > > > > > > > Pls take me off this VOTE list > > > > > > > > > > Best, > > > > > > > > > > Patrick Williams > > > > > > > > > > Sales Manager, UK & Ireland, Nordics & Israel > > > > > StorageOS > > > > > +44 (0)7549 676279 > > > > > patrick.williams@storageos.com > > > > > > > > > > 20 Midtown > > > > > 20 Proctor Street > > > > > Holborn > > > > > London WC1V 6NX > > > > > > > > > > Twitter: @patch37 > > > > > LinkedIn: linkedin.com/in/patrickwilliams4 < > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Flinked= in.com%2Fin%2Fpatrickwilliams4&data=3D02%7C01%7C%7C9b12ec4ce9ae4454db8a= 08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C63680110125299409= 2&sdata=3DipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3D&reserv= ed=3D0 > > > > > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fslack= .storageos.com%2F&data=3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%= 7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata= =3DhxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3D&reserved=3D0 > > > > > > > > > > > > > > > > > > > > On 03/12/2018, 17:34, "Guozhang Wang" wrote: > > > > > > > > > > Hello Boyang, > > > > > > > > > > I've browsed through the new wiki and there are still a couple of > > > > > minor > > > > > things to notice: > > > > > > > > > > 1. RemoveMemberFromGroupOptions seems not defined anywhere. > > > > > > > > > > 2. LeaveGroupRequest added a list of group instance id, but still > > > > > keep the > > > > > member id as a singleton; is that intentional? I think to make > > > > the > > > > > protocol > > > > > consistent both member id and instance ids could be plural. > > > > > > > > > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm > > > > > wondering if we > > > > > can defer adding this while just add the corresponding calls of > > > > the > > > > > LeaveGroupRequest inside Streams until we have used it in > > > > > production and > > > > > hence have a better understanding on how flexible or extensible > > > > if > > > > > we want > > > > > to add any cmd tools. The rationale is that if we do not > > > > > necessarily need > > > > > it now, we can always add it later with a more think-through API > > > > > design, > > > > > but if we add the tool in a rush, we may need to extend or modify > > > > > it soon > > > > > after we realize its limits in operations. > > > > > > > > > > Otherwise, I'm +1 on the proposal. > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen > > > > > wrote: > > > > > > > > > > > Hey community friends, > > > > > > > > > > > > after another month of polishing, KIP-345< > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bsta= tic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data= =3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaa= aaaaaaaa%7C1%7C0%7C636801101252994092&sdata=3DT4i7L1i0nIeHrrjTeLOOgYKsf= zfNEMGDhTazvBEZbXw%3D&reserved=3D0 > > > > > > > > > > > > design is ready for vote. Feel free to add your comment on the > > > > > discussion > > > > > > thread or here. > > > > > > > > > > > > Thanks for your time! > > > > > > > > > > > > Boyang > > > > > > ________________________________ > > > > > > From: Boyang Chen > > > > > > Sent: Friday, November 9, 2018 6:35 AM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: [VOTE] KIP-345: Introduce static membership protocol > > > > to > > > > > reduce > > > > > > consumer rebalances > > > > > > > > > > > > Hey all, > > > > > > > > > > > > > > > > > > thanks so much for all the inputs on KIP-345 so far. The > > > > > original proposal > > > > > > has enhanced a lot with your help. To make sure the > > > > > implementation go > > > > > > smoothly without back and forth, I would like to start a vote > > > > on > > > > > the final > > > > > > design agreement now: > > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=3D02%7C01%7C%7C9= b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%= 7C636801101252994092&sdata=3Dg4%2BMXKpkiQLZXg5HJWfJhw1kc1PbDNwyiX9zkREV= qGE%3D&reserved=3D0 > > > > < > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bsta= tic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data= =3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaa= aaaaaaaa%7C1%7C0%7C636801101252994092&sdata=3DT4i7L1i0nIeHrrjTeLOOgYKsf= zfNEMGDhTazvBEZbXw%3D&reserved=3D0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances= < > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bsta= tic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data= =3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaa= aaaaaaaa%7C1%7C0%7C636801101252994092&sdata=3DT4i7L1i0nIeHrrjTeLOOgYKsf= zfNEMGDhTazvBEZbXw%3D&reserved=3D0 > > > > > > > > > > > > > > > > > > > KIP-345: Introduce static membership protocol to reduce ...< > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fcwiki= .apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bsta= tic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data= =3D02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaa= aaaaaaaa%7C1%7C0%7C636801101252994092&sdata=3DT4i7L1i0nIeHrrjTeLOOgYKsf= zfNEMGDhTazvBEZbXw%3D&reserved=3D0 > > > > > > > > > > > > > cwiki.apache.org > > > > > > For stateful applications, one of the biggest performance > > > > > bottleneck is > > > > > > the state shuffling. In Kafka consumer, there is a concept > > > > called > > > > > > "rebalance" which means that for given M partitions and N > > > > > consumers in one > > > > > > consumer group, Kafka will try to balance the load between > > > > > consumers and > > > > > > ideally have ... > > > > > > > > > > > > > > > > > > Let me know if you have any questions. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Boyang > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -Regards, > > > > Mayuresh R. Gharat > > > > (862) 250-7125 > > > > > > > > > > > > > -- > > > Best, > > > Stanislav > > > > > > > > -- > -- Guozhang > --00000000000098eb47057eaa8bda--