From dev-return-93759-archive-asf-public=cust-asf.ponee.io@kafka.apache.org Wed May 2 21:11:52 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8121E18065D for ; Wed, 2 May 2018 21:11:51 +0200 (CEST) Received: (qmail 75403 invoked by uid 500); 2 May 2018 19:11:50 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 75391 invoked by uid 99); 2 May 2018 19:11:49 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 May 2018 19:11:49 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id EA624C0354 for ; Wed, 2 May 2018 19:11:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.879 X-Spam-Level: ** X-Spam-Status: No, score=2.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id lWmeWmojt1TQ for ; Wed, 2 May 2018 19:11:43 +0000 (UTC) Received: from mail-wm0-f47.google.com (mail-wm0-f47.google.com [74.125.82.47]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E03C95F1F0 for ; Wed, 2 May 2018 19:11:42 +0000 (UTC) Received: by mail-wm0-f47.google.com with SMTP id o78so26347249wmg.0 for ; Wed, 02 May 2018 12:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=RyfAYAcF2HYSK8h/2QwKacTVU1EV27xAoVci/g2cxTM=; b=A6L+8n/UzKFBzA+UoqciB7c/WccNDW2nGAwY84/rm46HM0Nye2sz3xFNb9udG+AEyg 7MJBjjmDpZ4d0091ymWl1NHAkUse9PzDeI1JWPQok6029vlslWoDjUZFEcoxfG5pyKAa DjDgOMjjrD5zUxb75U9sGGAUS7nQrZwWons33mKy1pYc4cHW1ca/yAMKZieky6M88gv0 BDbM2syTs8y3frFmrjFpx5WdEMOSmhWBwQfBC6nrEWyG1nNGcLWdbEP/Tj5jppciFaDl 3wZeEsyibVfJUHgwH4pnWComKRpSS5s18QpqZzWzKkQ9Exd8+CSUaM2TzfUJ/C0yp39l roWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=RyfAYAcF2HYSK8h/2QwKacTVU1EV27xAoVci/g2cxTM=; b=rsp07bpV39ZEwLF7Ck1NGbiNdZWbKBvtU5xsayZZOHnC4KUOkfOZrd100D+hebO4AG yzh/iqFXzJjWFBnqCWQI0zHoQ0N/+p3djgtoGmKgRJuEKbfF0t5cnpRHlP32UPpR+lW+ T8bESclm0ViSCqbLHpQV6pQLcXRBVMkm/VdTI2PGvcLE96vFIPsZuLxvDzkGcEA2tRpq MFeG+PLgrabBJxSZ8Q/WIp7hr62Gehpf3XY0eiOQJeE/S5S+lmNjrld+ibe6/y+FNuXw Isu8oIELZ0WtYxKmVSrxFV/Z1dASG2DfmFer5KcrPumfQpGXFWKnx2Mzuh/I+V0gP+6V B7PQ== X-Gm-Message-State: ALQs6tDS65evrm9j2A2rmx1XNPhJXbIhv8qa66XnbIfeYwHsQMyysduS ZHufowd50n2qV/VQ3zJdI78OQY8HAs/W2gM1g03ERg== X-Google-Smtp-Source: AB8JxZoAUzUKddGM9txN9iRH01OHip2l3QE1PRb0m4hp32nxOe6IFWBU+fTTKM0WYwj4F/bUFAJ7m0QiVueOgi+owBg= X-Received: by 2002:a50:a662:: with SMTP id d89-v6mr27934036edc.296.1525288301683; Wed, 02 May 2018 12:11:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.80.185.69 with HTTP; Wed, 2 May 2018 12:11:41 -0700 (PDT) In-Reply-To: <1525282083.1292021.1358514432.73192C6B@webmail.messagingengine.com> References: <12c91b3f-5dc6-0a64-8e9e-fe8d11aa6bbb@oss.nttdata.com> <1524501568.2561264.1347810648.0783E863@webmail.messagingengine.com> <54cd74b0-9159-ff9b-7298-15feaeea449c@oss.nttdata.com> <1524589614.3835596.1349260760.758F361A@webmail.messagingengine.com> <238c7fd9-559b-2c2a-b15b-b3aca76a93f4@oss.nttdata.com> <1525184254.2331997.1356927872.5BEA8589@webmail.messagingengine.com> <1525282083.1292021.1358514432.73192C6B@webmail.messagingengine.com> From: Manikumar Date: Thu, 3 May 2018 00:41:41 +0530 Message-ID: Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's commands To: dev Content-Type: multipart/alternative; boundary="00000000000066e74d056b3ddb24" --00000000000066e74d056b3ddb24 Content-Type: text/plain; charset="UTF-8" Hi Colin, Thanks for explanation. It's definitely useful to have --version flag. kafka-broker-api-versions.sh gives the API versions, not Kafka release version. Is not easy to figure out release version from API versions. Currently release version is available via metric/JMX. If required, we can implement this in future. Thanks, On Wed, May 2, 2018 at 10:58 PM, Colin McCabe wrote: > Hi Manikumar, > > We already have a tool for getting the Kafka broker API versions, > "./bin/kafka-broker-api-versions.sh". It was added as part of KIP-97. > > What Saski is proposing here is having a way of getting the version of > locally installed Kafka software, which may be different from the server > version. Many pieces of software offer a --version flag, and it's always > understood to refer to the local version of the software, not a version > running somewhere else. The user has no way to get this information now, > other than perhaps trying to look at he names of jar files. > > cheers, > Colin > > On Tue, May 1, 2018, at 08:20, Manikumar wrote: > > I assume the intent of the KIP to find out the Kafka broker version. In > > this case, maybe we should expose > > version using a Kafka request. This will help the remote scripts/tools to > > query the Kafka version. > > scripts (kafka-topics.sh, kafka-configs.sh, etc..) may run from remote > > machines and may use > > older Kafka versions. In this case, current proposal prints on the older > > version. > > > > On Tue, May 1, 2018 at 7:47 PM, Colin McCabe wrote: > > > > > Thanks, Sasaki. > > > > > > Colin > > > > > > On Sat, Apr 28, 2018, at 00:55, Sasaki Toru wrote: > > > > Hi Colin, Jason, > > > > > > > > Thank you for your beneficial comment. > > > > I have updated my Pull Request to show git commit hash in version > > > > information.> In my current Pull Request, we cat get the result such > > > below: > > > > > > > > $ bin/kafka-topics.sh --version > > > > (snip) > > > > 2.0.0-SNAPSHOT (Commit:f3876cd9617faf7e) > > > > > > > > > > > > I have also updated to accept double-dash for this option (-- > > > > version) only.> > > > > > > > > Many thanks, > > > > Sasaki > > > > > > > > > From: Jason Gustafson > > > > > Date: 2018-04-25 9:42 GMT+09:00 > > > > > Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's > > > > > commands> > To: dev > > > > > > > > > > > > > > > +1 on adding the git commit id to the output. We often encounter > > > > > environments which are testing off of trunk or have modifications > on > > > > > top of> > an existing release. > > > > > > > > > > -Jason > > > > > > > > > > On Tue, Apr 24, 2018 at 10:06 AM, Colin McCabe > > > > > > wrote:> > > > > > >> On Tue, Apr 24, 2018, at 05:36, Sasaki Toru wrote: > > > > >>> Hi Jason, Colin, > > > > >>> > > > > >>> Thank you for your comment, and I'm sorry for late reply. > > > > >>> > > > > >>> > we refactored all of the tools so that we could use a common > > > > >>> > set of> >>> base options, > > > > >>> > it might be a little annoying to have to continue supporting > > > > >>> > both> >>> variations. > > > > >>> > > > > >>> I feel KIP-14 is good idea (I'm sorry, I didn't know about > > > > >>> it), and> >>> I think it's no longer necessary both single-dash > and > > > double- > > > > >>> dash are> >>> supported when this KIP will be accepted, as you > said. > > > > >>> I revise my Pull Request to support single-dash option only. > > > > >>> > > > > >>> Some my code in kafka-run-class.sh will become unnecessary when > > > > >>> KIP-14> >>> is accepted. > > > > >>> But I will keep this as temporary, because I seem that it's > useful> > > > >>> before KIP-14 accepted. > > > > >>> > > > > >>> > > > > >>> > Can you give a little more detail about what would be > > > > >>> > displayed when> >>> the version command was used? > > > > >>> > > > > >>> As Ismael said, the version string is got from > > > > >>> AppInfoParser#getVersion.> >>> > > > > >>> In my Pull Request, we can get the result such as below:: > > > > >>> > > > > >>> $ bin/kafka-topics.sh --version > > > > >>> (snip) > > > > >>> Kafka 1.2.0-SNAPSHOT > > > > >> Hi Sasaki, > > > > >> > > > > >> Thanks for the info. Can you add this to the KIP? > > > > >> > > > > >> Also, I really think we should include the git hash. > > > > >> > > > > >> best, > > > > >> Colin > > > > >> > > > > >> > > > > >>> > > > > >>> Many thanks, > > > > >>> Sasaki > > > > >>> > > > > >>> > > > > >>>> From: Ismael Juma > > > > >>>> Date: 2018-04-24 3:44 GMT+09:00 > > > > >>>> Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's > > > > >>>> commands> >>>> To: dev > > > > >>>> > > > > >>>> > > > > >>>> FYI, the injection via the build process that is mentioned here > > > > > already > > > > >>>> happens. See AppInfoParser. > > > > >>>> > > > > >>>> Ismael > > > > >>>> > > > > >>>> On Mon, Apr 23, 2018 at 9:39 AM, Colin McCabe > > > > >>>> > >> wrote: > > > > >>>>> Hi Sasaki, > > > > >>>>> > > > > >>>>> Thanks for the KIP. I think a version flag is a good idea. > > > > >>>>> > > > > >>>>> Can you give a little more detail about what would be displayed > > > > >>>>> when> >> the > > > > >>>>> version command was used? > > > > >>>>> > > > > >>>>> We clearly want the version number, but we probably also want > to > > > > >>>>> know> >> if > > > > >>>>> this is an official release, or a random SNAPSHOT from a > branch. > > > > >>>>> If> >> this > > > > >>>>> is a release candidate, we probably want the RC number as well, > > > > >>>>> like> >>>>> "1.1-rc3" We also want a git hash. This can be > > > injected by the> > build > > > > >>>>> process. In the case of an official release, where the source > > > > >>>>> code> >> is not > > > > >>>>> under git, we can pull it from a file. > > > > >>>>> > > > > >>>>> For example, hadoop's version output looks like this: > > > > >>>>> > > > > >>>>> > cmccabe@aurora:~/Downloads/hadoop-2.8.3> ./bin/hadoop > > > > >>>>> > version> >>>>> > Hadoop 2.8.3 > > > > >>>>> > Subversion > > > > >>>>> > https://git-wip-us.apache.org/repos/asf/hadoop.git -r> > >>>>> > > > b3fe56402d908019d99af1f1f4fc65cb1d1436a2 > > > > >>>>> > Compiled by jdu on 2017-12-05T03:43Z > > > > >>>>> > Compiled with protoc 2.5.0 > > > > >>>>> > From source with checksum 9ff4856d824e983fa510d3f843e3f1 > 9d> > > > >>>>> > This command was run using /home/cmccabe/Downloads/ > > > > >>>>> hadoop-2.8.3/share/hadoop/common/hadoop-common-2.8.3.jar > > > > >>>>> > > > > >>>>> (The "subversion" line here is a little weird -- it now refers > > > > >>>>> to> >> git, not > > > > >>>>> svn) > > > > >>>>> > > > > >>>>> On Wed, Apr 11, 2018, at 13:58, Jason Gustafson wrote: > > > > >>>>>> Hey Sasaki, > > > > >>>>>> > > > > >>>>>> Yeah, I don't feel too strongly about only supporting -- > > > > >>>>>> version. I> >> agree > > > > >>>>> it > > > > >>>>>> may help discoverability given the current approach. On the > > > > >>>>>> other> >> hand, > > > > >>>>> if > > > > >>>>>> we refactored all of the tools so that we could use a common > > > > >>>>>> set of> >> base > > > > >>>>>> options, it might be a little annoying to have to continue > > > > > supporting > > > > >>>>> both > > > > >>>>>> variations. For example, tool standardization was proposed in > > > > >>>>>> KIP-14> >> and > > > > >>>>>> I'm still holding out hope that someone will have time to pick > > > > >>>>>> this> >> work > > > > >>>>>> back up. It's always easier to add an option than remove one, > > > > >>>>>> so I'm> >>>>>> slightly inclined to have only --version for > now. > > > What do you > > > > >>>>>> think?> >>>>> The double dash version is more consistent with > how > > > our other > > > > >>>>> flags> >> work. > > > > >>>>> In general, I feel that if --version is supported, --help > should > > > > >>>>> say> >> so. > > > > >>>>> best, > > > > >>>>> Colin > > > > >>>>> > > > > >>>>> > > > > >>>>>> Thanks, > > > > >>>>>> Jason > > > > >>>>>> > > > > >>>>>> On Tue, Apr 10, 2018 at 12:00 AM, Sasaki Toru < > > > > >> sasakitoa@oss.nttdata.com > > > > >>>>>> wrote: > > > > >>>>>> > > > > >>>>>>> Hi Jason > > > > >>>>>>> > > > > >>>>>>> Thank you for helpful comments. I updated wiki based on your > > > > > advice. > > > > >>>>>>> I thought this option was relatively common and making > > > > >>>>>>> maintenance> >>>> easy > > > > >>>>>>> was also important. > > > > >>>>>>> However, as you said, it is not good that version option > won't > > > > >>>>>>> be> >>>>> shown up > > > > >>>>>>> in help description. > > > > >>>>>>> > > > > >>>>>>> I thought accepting both single-dash and double-dash will > help > > > > >>>>>>> to> >> find > > > > >>>>>>> this option. > > > > >>>>>>> In my approach this option won't be showed, but most of > > > > >>>>>>> software> >> which > > > > >>>>> has > > > > >>>>>>> this option accepts either single-dash or double-dash. > > > > >>>>>>> I guess it doesn't need to support both if we take another > > > > >>>>>>> way.> >>>>>>> > > > > >>>>>>> > > > > >>>>>>> Thanks > > > > >>>>>>> > > > > >>>>>>> @Ted Yeah, you're right. Sorry about the confusion. > > > > >>>>>>>> Since we're here, I think this KIP is a nice improvement. > > > > >>>>>>>> It's> >>>>> definitely > > > > >>>>>>>> nice to have an easy way to check the version. That said, do > > > > >>>>>>>> we> >>>> really > > > > >>>>>>>> need > > > > >>>>>>>> to support both `-version` and `--version`? The latter is > > > > >> consistent > > > > >>>>> with > > > > >>>>>>>> our current tools. > > > > >>>>>>>> > > > > >>>>>>>> Also, I think the approach we've taken is basically to build > > > > >>>>>>>> the> >>>>> --version > > > > >>>>>>>> functionality into the bash script. This is nice because it > > > > >>>>>>>> saves> > a > > > > >>>>> lot of > > > > >>>>>>>> work to update the commands individually and we don't need > to > > > > >>>>>>>> do> >>>>> anything > > > > >>>>>>>> when we add new tools. The downside is that `--version` > won't > > > > >>>>>>>> show> >> up > > > > >>>>> as > > > > >>>>>>>> an > > > > >>>>>>>> option in any of the --help output. Not sure if that is too > > > > >>>>>>>> big of> >> a > > > > >>>>>>>> problem, but maybe worth mentioning this in the rejected > > > > >> alternatives > > > > >>>>>>>> section. > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> -Jason > > > > >>>>>>>> > > > > >>>>>>>> On Wed, Apr 4, 2018 at 9:42 AM, Ted Yu >> > > > >> wrote: > > > > >>>>>>>> Jason: > > > > >>>>>>>>> Maybe your reply was intended for another KIP ? > > > > >>>>>>>>> > > > > >>>>>>>>> KIP-278 is about adding version option, not timeout. > > > > >>>>>>>>> > > > > >>>>>>>>> Cheers > > > > >>>>>>>>> > > > > >>>>>>>>> On Wed, Apr 4, 2018 at 9:36 AM, Jason Gustafson < > > > > >> jason@confluent.io> > > > > >>>>>>>>> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>> Hi Sasaki, > > > > >>>>>>>>>> Thanks for the KIP. I think the timeout controls the > > > > >>>>>>>>>> maximum> >>>> allowed > > > > >>>>>>>>> time > > > > >>>>>>>>> that the consumer will block for the next record. Maybe > the> > > > >> meaning > > > > >>>>>>>>> would > > > > >>>>>>>>> be clearer with the more concise name `--timeout`? That > also > > > > >>>>>>>>> fits> >>>>> with > > > > >>>>>>>>> the > > > > >>>>>>>>> > > > > >>>>>>>>>> old consumer which overrides the `consumer.timeout.ms` > > > > >>>>>>>>>> property.> >>>>>>>>>> > > > > >>>>>>>>>> By the way, it seems like the default value was > > > > >>>>>>>>>> intentionally> > set > > > > >>>>> low > > > > >>>>>>>>> for > > > > >>>>>>>>> both the old and new consumers, but I'm not sure of the > > > > >>>>>>>>> reason.> > We > > > > >>>>> could > > > > >>>>>>>>>> leave the default as it is if we want to be safe, but > > > > >>>>>>>>>> increasing> >> it > > > > >>>>>>>>> seems > > > > >>>>>>>>> ok to me. Perhaps we could start a little lower, though, > say > > > > >>>>>>>>> 10> >>>>> seconds? > > > > >>>>>>>>> In > > > > >>>>>>>>> > > > > >>>>>>>>>> any case, we should make it clear to the user that the > > > > >>>>>>>>>> timeout> >> was > > > > >>>>>>>>> reached. > > > > >>>>>>>>> > > > > >>>>>>>>>> It's surprising to see only the incomplete reported > results> > > > >>>>> following a > > > > >>>>>>>>>> timeout. > > > > >>>>>>>>>> > > > > >>>>>>>>>> Thanks, > > > > >>>>>>>>>> Jason > > > > >>>>>>>>>> > > > > >>>>>>>>>> On Wed, Apr 4, 2018 at 4:37 AM, Sasaki Toru < > > > > >>>>> sasakitoa@oss.nttdata.com> > > > > >>>>>>>>>> wrote: > > > > >>>>>>>>>> > > > > >>>>>>>>>> Hello everyone, > > > > >>>>>>>>>>> I would like to start a discussion for KIP 278. Cloud > you> > > > > please > > > > >>>>> give > > > > >>>>>>>>>>> comments and advice ? > > > > >>>>>>>>>>> > > > >>>>>>>>>>> 278+-> >>>>>>>>>>> +Add+version+option+to+Kafka%2 > > > 7s+commands> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> JIRA ticket and Pull Request are bellow: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Many thanks, > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Sasaki > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> -- > > > > >>>>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA > > > > >>>>>>>>>>> CORPORATION> >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>> -- > > > > >>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION > > > > >>>>>>> > > > > >>>>>>> > > > > >>> -- > > > > >>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION > > > > >>> > > > > > > > > -- > > > > Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION > > > > > > > > > > > --00000000000066e74d056b3ddb24--