From dev-return-100168-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Mon Dec 3 18:34:12 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id A4C9C180645 for ; Mon, 3 Dec 2018 18:34:11 +0100 (CET) Received: (qmail 483 invoked by uid 500); 3 Dec 2018 17:34:10 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 470 invoked by uid 99); 3 Dec 2018 17:34:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Dec 2018 17:34:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 297B918BEC6 for ; Mon, 3 Dec 2018 17:34:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.778 X-Spam-Level: * X-Spam-Status: No, score=1.778 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id JeZEM9Wa7U1g for ; Mon, 3 Dec 2018 17:34:07 +0000 (UTC) Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 69B4360D3B for ; Mon, 3 Dec 2018 17:27:38 +0000 (UTC) Received: by mail-ot1-f45.google.com with SMTP id 40so12361882oth.4 for ; Mon, 03 Dec 2018 09:27:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=ICAqjCPUVjC2pbT/548YTHuWs8LucN7Gd10O0MW1p0I=; b=GKF2NCw9xk2xld0ZkqUOSMuQRIb/0fCbiuKEyebLKs2SpDYO+vSexpw/6tS/Vb8DjE pjRWVqqP3AZi86suB01IxT30+Gxo/bwgqbcpoQtINCe7h+puravL2+rGJt3zoseRI0VE yyVEGsSYcXX0/VYA4jgmSeyNekxin3m57DmjQ2CydUXmtWqzmUjMojsY42iI8w38QSEI UuTP/jRFrHZPnq7q5uvALm6oQteJGCw0Z6dXZWrqX/tj2zm9+sbH+A+VPEkLNPXO4TrL 6idTvB1GWqFmpFjQLwKcZ69naqTeOUV/irzMfcybLk1oJtoty0lgrceCDX5Nnjzq/W05 8LjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=ICAqjCPUVjC2pbT/548YTHuWs8LucN7Gd10O0MW1p0I=; b=TK//ulS84saMHa9chNs0LEs1yFKpz/4rBuwYurFsz5I9e4mR2Gfyj8fJBIXruBWFu7 uWBx69DEkdsjlpFuWJK/+0Bq5pMTTuvqvb18QVgEnyjCGMlxTdGNmtb9blo15OhRx54J g06tX1opu8qb8uhz7GgvJm0ea+9AXjY1c28495ynZmuCnwsBHr5hl9BPOcPXM4rCY4G/ BWNg4EUFDpZeGZrrIhuVqWbKNjoH6k9/iO7TTEPtr+Nmexh6+6TsbhneoBDFmmIXXjhD 6fG/Xyp3gNZtXvwz4Bjyado+nICMNGQ8b0nmYEPQWZwn8y+Rp6hAQKi0KXYSgehySfGD tdKg== X-Gm-Message-State: AA+aEWYE89uTn9BYy9I1g1XxoW/ot8IR6DM9MvhmD9dXJFaOgchnZBUO meKmQiA9hxmCAr8qtOOgatSyvIiKNwLQRpsQXe/7sZhI X-Google-Smtp-Source: AFSGD/Wer+/EKJ/VC/PCh40f4urwkTvBapr1N+9CSCEeuYqotUVVV0C5v/I9ZjUzXCmwLMnmyRv3xqVjr5R3QfaR6bE= X-Received: by 2002:a9d:7a8c:: with SMTP id l12mr11028557otn.335.1543858056774; Mon, 03 Dec 2018 09:27:36 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Guozhang Wang Date: Mon, 3 Dec 2018 09:27:25 -0800 Message-ID: Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances To: dev Content-Type: multipart/alternative; boundary="0000000000000ec86c057c217795" --0000000000000ec86c057c217795 Content-Type: text/plain; charset="UTF-8" Hello Boyang, I've browsed through the new wiki and there are still a couple of minor things to notice: 1. RemoveMemberFromGroupOptions seems not defined anywhere. 2. LeaveGroupRequest added a list of group instance id, but still keep the member id as a singleton; is that intentional? I think to make the protocol consistent both member id and instance ids could be plural. 3. About the *kafka-remove-member-from-group.sh *tool, I'm wondering if we can defer adding this while just add the corresponding calls of the LeaveGroupRequest inside Streams until we have used it in production and hence have a better understanding on how flexible or extensible if we want to add any cmd tools. The rationale is that if we do not necessarily need it now, we can always add it later with a more think-through API design, but if we add the tool in a rush, we may need to extend or modify it soon after we realize its limits in operations. Otherwise, I'm +1 on the proposal. Guozhang On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen wrote: > Hey community friends, > > after another month of polishing, KIP-345< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances> > design is ready for vote. Feel free to add your comment on the discussion > thread or here. > > Thanks for your time! > > Boyang > ________________________________ > From: Boyang Chen > Sent: Friday, November 9, 2018 6:35 AM > To: dev@kafka.apache.org > Subject: [VOTE] KIP-345: Introduce static membership protocol to reduce > consumer rebalances > > Hey all, > > > thanks so much for all the inputs on KIP-345 so far. The original proposal > has enhanced a lot with your help. To make sure the implementation go > smoothly without back and forth, I would like to start a vote on the final > design agreement now: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances > > > > 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances > > > > KIP-345: Introduce static membership protocol to reduce ...< > https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances > > > cwiki.apache.org > For stateful applications, one of the biggest performance bottleneck is > the state shuffling. In Kafka consumer, there is a concept called > "rebalance" which means that for given M partitions and N consumers in one > consumer group, Kafka will try to balance the load between consumers and > ideally have ... > > > Let me know if you have any questions. > > > Best, > > Boyang > > -- -- Guozhang --0000000000000ec86c057c217795--