From dev-return-100744-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Thu Jan 3 22:42:20 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BACD2180608 for ; Thu, 3 Jan 2019 22:42:19 +0100 (CET) Received: (qmail 94603 invoked by uid 500); 3 Jan 2019 21:42:18 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 94591 invoked by uid 99); 3 Jan 2019 21:42:17 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jan 2019 21:42:17 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 42392C0162 for ; Thu, 3 Jan 2019 21:42:17 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.798 X-Spam-Level: * X-Spam-Status: No, score=1.798 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 4KrpAzjeS_p6 for ; Thu, 3 Jan 2019 21:42:15 +0000 (UTC) Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 2E46360E75 for ; Thu, 3 Jan 2019 21:42:15 +0000 (UTC) Received: by mail-ot1-f52.google.com with SMTP id 81so30584983otj.2 for ; Thu, 03 Jan 2019 13:42:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=mbBmKHjkL/SYFp32hf5n9OhN6frjTlVvkqiQLapWT4I=; b=DNp3GvMwF5WHnWQZTme8kV158q4no274hhBCSO6OfYELWNCJItRB7D/tyjs7E102eq H3Z54dzFR5T0Ew9lnOyf62wKF5JHdaXAsD5hLuq2eX02R8hXBr2YgCvn/9FQ02cqaCiu HxYGzOK4k1ba0Ng0qQbtCkgYt7N9oT7Lms6wlGr3HuI07+xSeKSr37ao1j7zb+2tixwg YFhhBY5vGK1js2BP/OnAQw4ilS3NxWfmI7ZDdaiIpx6G5xjtfgtLdEdLF/sD97qmyqkG VUL+DSWUzl1Uefvz06SijUYmb/6Qv9MpWZJVFdK5+fxEeaOcV+fuIEnwRb0C2baj7O/p xkSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=mbBmKHjkL/SYFp32hf5n9OhN6frjTlVvkqiQLapWT4I=; b=Gl99bOcrZlANUw3zuQpVe8QqZA2q8KMgX2rmmXZmtx2kDe4O+w+Vifo//jg/ApMalb aMBlR6Jqyo2S3k0Oogwn+4lFGzadLCL8KBjA0BjDH8euMuNGgBbHMSI1mYT+vzWKpYjK HJPlmvJ6cPWKLTk0ZAwKRAS5ETV9GhjUgi4lpWL+IBSGG1Ta1bvhOWMLcYEttISERrQU IWj5/b/tlGkAbpwI5PPCu4YwQljawOhtZMeVxCjHaXZKw5uU8D3d3mnXyun8IavHUDM1 AmVeGN5NgaNc0JQNCmSoxpuH/DdSUTSs2MsbdKGU8vf6KDWx1/L/L8aqbce20hEWseRG Jsbg== X-Gm-Message-State: AJcUukeZaKXPQZHc67eXiZ1+0+FMdESEC7ELqPcz5R+KKMoW2jMj+7TY OYxgrjRRz2+MTPrZH1ikyJsjuHkvLg9ChsZevOCETQ== X-Google-Smtp-Source: ALg8bN6FJvPRiapo959ZzNhTPulZKG7DuCqDJxmApTgTL3ASV/I1XmfdO3gGYITa5eEdR5Uq7gUnj7wixzTUEighWgc= X-Received: by 2002:a9d:7e87:: with SMTP id m7mr36567225otp.225.1546551727439; Thu, 03 Jan 2019 13:42:07 -0800 (PST) MIME-Version: 1.0 References: <7d84858a-e012-3bb8-3c54-9cd2362b4178@confluent.io> <667ee58f-9a39-1071-2137-82ab42fbe569@confluent.io> <5469d09b-75ae-a48e-ff91-c2743c764e9f@confluent.io> In-Reply-To: <5469d09b-75ae-a48e-ff91-c2743c764e9f@confluent.io> From: Richard Yu Date: Thu, 3 Jan 2019 13:41:56 -0800 Message-ID: Subject: Re: [DISCUSS] KIP-262 Metadata should include the number of state stores for task To: dev@kafka.apache.org Content-Type: multipart/alternative; boundary="0000000000005745b4057e94a256" --0000000000005745b4057e94a256 Content-Type: text/plain; charset="UTF-8" Hi Matthias, I guess this is no longer necessary. I am open to anything honestly. I suppose we should close it (if its not already). On Fri, Oct 19, 2018 at 11:06 AM Matthias J. Sax wrote: > Any thought on my last email about discarding this KIP? > > > -Matthias > > On 9/14/18 11:44 AM, Matthias J. Sax wrote: > > Hi, > > > > we recently had a discussion on a different ticket to reduce the size of > > the metadata we need to send: > > https://issues.apache.org/jira/browse/KAFKA-7149 > > > > It seems, that we actually don't need to include the number of stores in > > the metadata, but that we can compute the number of stores locally on > > each instance. > > > > With this insight, we should still try to exploit this knowledge during > > task assignment, however, this would be an internal change that does not > > require a KIP. Thus, I think that we can discard this KIP. > > > > Thoughts? > > > > > > -Matthias > > > > On 6/10/18 5:20 PM, Matthias J. Sax wrote: > >> Richard, > >> > >> KIP-268 got merged and thus this KIP is unblocked. > >> > >> I just re-read it and think it needs some updates with regard to the > >> upgrade path (ie, you should mention why upgrading is covered). > >> > >> It would also be useful to discuss how the store information is used > >> during assignment. Atm, the KIP only discussed that the information > >> should be added, but this is only half of the story from my point of > view. > >> > >> > >> -Matthias > >> > >> On 3/22/18 9:15 PM, Matthias J. Sax wrote: > >>> Hi Richard, > >>> > >>> with KIP-268 in place (should be accepted soon) the upgrade path is > >>> covered. Thus, you can update your KIP accordingly, referring to > KIP-268. > >>> > >>> Can you also update your KIP similar to KIP-268 to cover the old and > new > >>> metadata format? > >>> > >>> Thanks! > >>> > >>> -Matthias > >>> > >>> > >>> On 2/24/18 4:07 PM, Richard Yu wrote: > >>>> I didn't really get what "upgrade strategy" was at the time that > Guozhang > >>>> mentioned it, so I wrote the above dialogue from my first > understanding. I > >>>> changed it to "upgrade strategy must be provided". Currently, > however, I do > >>>> not have anything in mind to facilitate upgrading older Kafka > brokers. If > >>>> you have anything in mind, please let me know. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax < > matthias@confluent.io> > >>>> wrote: > >>>> > >>>>> Thanks a lot for this KIP. > >>>>> > >>>>> I am not sure what you mean by > >>>>> > >>>>>> which could potentially break older versions of Kafka brokers > >>>>> > >>>>> The metadata that is exchange, is not interpreted by the brokers. The > >>>>> problem with upgrading the metadata format affect only Kafka Streams > >>>>> instances. > >>>>> > >>>>> If we don't provide an upgrade strategy, changing the metadata format > >>>>> required to stop all running application instances, before the > instances > >>>>> can be restarted with the new code. However, this implies downtime > for > >>>>> an application and is thus not acceptable. > >>>>> > >>>>> > >>>>> -Matthias > >>>>> > >>>>> > >>>>> On 2/24/18 11:11 AM, Richard Yu wrote: > >>>>>> Hi all, > >>>>>> > >>>>>> I would like to discuss a KIP I've submitted : > >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>> 262%3A+Metadata+should+include+number+of+state+stores+for+task > >>>>>> > >>>>>> > >>>>>> Regards, > >>>>>> Richard Yu > >>>>>> > >>>>> > >>>>> > >>>> > >>> > >> > > > > --0000000000005745b4057e94a256--