From dev-return-92903-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Mon Apr 2 20:25:35 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 25C74180627 for ; Mon, 2 Apr 2018 20:25:34 +0200 (CEST) Received: (qmail 59873 invoked by uid 500); 2 Apr 2018 18:25:33 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 59843 invoked by uid 99); 2 Apr 2018 18:25:33 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2018 18:25:33 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id AA229C1312 for ; Mon, 2 Apr 2018 18:25:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.898 X-Spam-Level: * X-Spam-Status: No, score=1.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id iQUzU7Ik-Z0z for ; Mon, 2 Apr 2018 18:25:31 +0000 (UTC) Received: from mail-ot0-f176.google.com (mail-ot0-f176.google.com [74.125.82.176]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 268FA5F1F0 for ; Mon, 2 Apr 2018 18:25:31 +0000 (UTC) Received: by mail-ot0-f176.google.com with SMTP id m7-v6so16614368otd.1 for ; Mon, 02 Apr 2018 11:25:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=22u+NNa3iJTY8DRe0vSfQg5lxr2FNh5k/TfsoiKep60=; b=chVHpQxa7NBZanFAXD4k2zY3xBYjNfFxOmbIq2YlpQbkgB5kDGzBIQYafRVE01v7bc WFS6jvkMZHfoZYWIDWGru+nXPPOuPkPWvh10p5hQnB3lmEyFJ5ABbDIodaJc6ur2LUrO aeR5396AGObK/5LfYyxcD0GGwxZsj5A6V/U/ApXhDBWiAhM+dM4gZxG9vdPF1v0ClApV TL4A/jx1vyz1aZbm1iDiA1CSwRmTTpzqJ6CKF/SZwYeUbtKrRdvVoBgAmDwrnWzHZ59V rNdOuBaoUrys+mYEIJYnIdjtQrNAZo1CY5HYlr5q4Xnx0/+1IKWqO1g62EnAlFlbc+CX KJzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=22u+NNa3iJTY8DRe0vSfQg5lxr2FNh5k/TfsoiKep60=; b=LGgrfPMZbt32pa2k8MHIuIKj2DRb+NT2JbPetmQPizwwBChyC5oge6q011Hc4OgbWN ZCMBZlXlvFOANXHiEZDjvkRKonjcvYyD+Ic+JlmFCgD4AWZNEZWIjz87PB5AzmmMrx30 heQdsaQ0MU9KBE1Wp+w4gjrTrKlZ7Xw1IY47T9AODt/fDFQ+tTm6n5H1NFcrBP1VXs1U Gl+oeHrZvx2hOrXga+lHMs50/+hj0zoXHUqQIfC4Hpmr3fLzDPwvQ7OOZPZJBzTEwK3J 7D/9BE70iQNO0Iatsj6ohdEPU/IUdKgxAh9uMefh5qKXn/vOR22QGMkOMDKmd5p4uzcM CD8w== X-Gm-Message-State: ALQs6tBSU82xrzmsgwN6HjwnoTZG3HE/XY9Zqwg7E7rHRZsWTEd9SOAS MkrPxaWz+X7/PIODcFhPnhtsaoktbdgPUx++aGA= X-Google-Smtp-Source: AIpwx48ArhiPktSwdIwtAO54iaRrcdrZ4eVVQ851RmLb3DBLP3fNx3Him6zNqK8sTRXtdky08VeU/0LwHNOWClkAYJ8= X-Received: by 2002:a9d:738b:: with SMTP id j11-v6mr5699608otk.169.1522693530099; Mon, 02 Apr 2018 11:25:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.74.189.151 with HTTP; Mon, 2 Apr 2018 11:25:29 -0700 (PDT) In-Reply-To: References: <1466f50d-f847-d066-fc43-95253131439a@confluent.io> From: Guozhang Wang Date: Mon, 2 Apr 2018 11:25:29 -0700 Message-ID: Subject: Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers To: dev@kafka.apache.org Content-Type: multipart/alternative; boundary="000000000000f6a5af0568e1b60c" --000000000000f6a5af0568e1b60c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable @Matthias That's a good question: I think adding another config for the main consumer makes good tradeoffs between compatibility and new user convenience. Just to clarify, from user's pov on upgrade: 1) I'm already overriding some consumer configs, and now I want to override these values differently for restore consumers, I'd add one new line for the restore consumer prefix. 2) I'm already overriding some consumer configs, and now I want to NOT overriding them for restore consumers, I'd change my override from `consumer.X` to `main.consumer.X`. 3) I'm new and have not any consumer overrides, and now if I want to override some, I'd use `main.consumer`, `restore.consumer` for specific consumer types, and ONLY consider `consumer` for the ones that I want to apply universally. 4) I'm already overriding some consumer configs and I'm happy with what I get, I do not change anything. Guozhang On Mon, Apr 2, 2018 at 11:10 AM, Ted Yu wrote: > bq. to introduce one more prefix `main.consumer.` > > Makes sense. > > On Mon, Apr 2, 2018 at 11:06 AM, Matthias J. Sax > wrote: > > > Boyang, > > > > thanks a lot for the KIP. > > > > Couple of questions: > > > > > (MODIFIED) public Map getRestoreConsumerConfigs(final > > String clientId); > > > > You mean that the implementation/semantics of this method changes, but > > not the API itself? Might be good to add this as "comment style" instea= d > > of "(MODIFIED)" prefix. > > > > > By rewriting the getRestoreConsumerConfigs() function and adding the > > getGlobalConsumerConfigs() function, if one user uses > > restoreConsumerPrefix() or globalConsumerPrefix() when adding new > > configurations, the configs shall overwrite base consumer config. If no= t > > specified, restore consumer and global consumer shall share the same > config > > with base consumer. > > > > While this does make sense for backward compatibility, I am wonder if i= t > > makes the config "inheritance logic" (ie, hierarchy) too complex? We > > basically introduce a second level of overwrites. It might be simpler t= o > > not introduce this hierarchy with the cost to break backward > compatibility. > > > > For example, config `request.timeout.ms`: > > > > User sets `request.timeout.ms=3D` > > To change it for the main consumer, user also sets > > `consumer.request.timeout.ms=3D` > > > > If user only wants to change the config for main consumer, but not for > > global or restore consumer, user needs to add two more configs: > > > > `restore.consumer.request.timeout.ms=3D` > > and > > `global.consumer.request.timeout.ms=3D` > > > > to reset both back to the default config. IMHO, this is not an optimal > > user experience. Thus, it might be worth to change the semantics for > > `consumer.` prefix to only apply those configs to the main consumer. > > > > > > Not sure what other think what the better solution is (I am not sure by > > myself to be honest---just wanted to point it out and discuss the > > pros/cons for both). > > > > > > Another though would be, to introduce one more prefix `main.consumer.` > > -- using this, the existing `consumer.` prefix would apply to all > > consumers (keeping it's current semantics) while we have overwrites for > > all three consumers -- this allow to directly set `main.consumer` > > instead of `consumer` avoiding the weird pattern from my example above > > and preserves backward compatibility. Ie, if we introduce an hierarchy > > of overwrite, a "full" hierarchy might be better than a "partial" > > hierarchy. > > > > > > Looking forward to your thoughts. > > > > > > -Matthias > > > > > > On 4/1/18 5:55 PM, Guozhang Wang wrote: > > > Thanks for the KIP Boyang, the KIP looks good to me. > > > > > > For config values, we use underscore for keeping a single word; for > > config > > > keys, though, we do not use underscores or dashes. I'd suggest using > dots > > > to be consistent with others. > > > > > > > > > Otherwise I'm +1 on the KIP. > > > > > > > > > Guozhang > > > > > > > > > On Sun, Apr 1, 2018 at 10:56 AM, Ted Yu wrote: > > > > > >> Looks good overall. > > >> > > >> public static final String RESTORE_CONSUMER_PREFIX =3D > > "restore-consumer."; > > >> > > >> For other constants in StreamsConfig, underscore is used instead of > > dash. > > >> > > >> Cheers > > >> > > >> On Sun, Apr 1, 2018 at 9:38 AM, Boyang Chen > > wrote: > > >> > > >>> Hey friends, > > >>> > > >>> > > >>> I would like to start a discussion thread for KIP 276: > > >>> > > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >>> 276+Add+StreamsConfig+prefix+for+different+consumers > > >>> > > >>> > > >>> And pull request is here: > > >>> > > >>> https://github.com/apache/kafka/pull/4805 > > >>> > > >>> [https://avatars3.githubusercontent.com/u/5845561?s=3D400&v=3D4 > ] > >>> github.com/apache/kafka/pull/4805> > > >>> > > >>> KAFKA-6657: Add StreamsConfig prefix for different consumers by > > abbccdda > > >> =C2=B7 > > >>> Pull Request #4805 =C2=B7 apache/kafka > >>> com/apache/kafka/pull/4805> > > >>> github.com > > >>> This pull request is for jira 6657. The KIP proposal is here Added > unit > > >>> tests for new getGlobalConsumerConfigs API and make sure existing > > restore > > >>> consumer tests are passing. > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> Thanks, > > >>> > > >>> Boyang > > >>> > > >> > > > > > > > > > > > > > > --=20 -- Guozhang --000000000000f6a5af0568e1b60c--