kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@apache.org>
Subject Re: [DISCUSS] KIP-278: Add version option to Kafka's commands
Date Wed, 09 May 2018 17:53:51 GMT
+1. Thanks, Sasaki.

Colin

On Wed, May 9, 2018, at 09:15, Jason Gustafson wrote:
> Hi Sasaki,
> 
> Thanks for the update. The KIP looks good to me. I'd suggest moving to a
> vote.
> 
> Thanks,
> Jason
> 
> On Mon, May 7, 2018 at 7:08 AM, Sasaki Toru <sasakitoa@oss.nttdata.com>
> wrote:
> 
> > Hi Manikumar, Colin,
> >
> > Thank you for your comment.
> >
> > As Colin said, I proposed adding an option to show version information of
> > "local" tool,
> > because many software have this option and I think Kafka also needs this
> > one.
> >
> > As you said, the function to show remote Kafka version is useful,
> > but I think it is better to create new KIP because this function has some
> > points which should be considered.
> >
> > If you have any better ideas, could you please tell us?
> >
> >
> > Many thanks,
> > Sasaki
> >
> > From: Manikumar <manikumar.reddy@gmail.com>
> >> Date: 2018-05-03 4:11 GMT+09:00
> >>
> >> Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's commands
> >> To: dev <dev@kafka.apache.org>
> >>
> >>
> >> Hi Colin,
> >>
> >> Thanks for explanation. It's definitely useful to have  --version flag.
> >>
> >> kafka-broker-api-versions.sh gives the API versions, not Kafka release
> >> version.
> >> Is not easy to figure out release version from API versions. Currently
> >> release version is available via metric/JMX.
> >> If required, we can implement this in future.
> >>
> >>
> >> Thanks,
> >>
> >> On Wed, May 2, 2018 at 10:58 PM, Colin McCabe <cmccabe@apache.org> wrote:
> >>
> >> Hi Manikumar,
> >>>
> >>> We already have a tool for getting the Kafka broker API versions,
> >>> "./bin/kafka-broker-api-versions.sh".  It was added as part of KIP-97.
> >>>
> >>> What Saski is proposing here is having a way of getting the version of
> >>> locally installed Kafka software, which may be different from the server
> >>> version.  Many pieces of software offer a --version flag, and it's always
> >>> understood to refer to the local version of the software, not a version
> >>> running somewhere else.  The user has no way to get this information now,
> >>> other than perhaps trying to look at he names of jar files.
> >>>
> >>> cheers,
> >>> Colin
> >>>
> >>> On Tue, May 1, 2018, at 08:20, Manikumar wrote:
> >>>
> >>>> I assume the intent of the KIP to find out the Kafka broker version.
 In
> >>>> this case, maybe we should expose
> >>>> version using a Kafka request. This will help the remote scripts/tools
> >>>>
> >>> to
> >>
> >>> query the Kafka version.
> >>>> scripts (kafka-topics.sh, kafka-configs.sh, etc..)  may run from remote
> >>>> machines  and may use
> >>>> older Kafka versions. In this case, current proposal prints on the older
> >>>> version.
> >>>>
> >>>> On Tue, May 1, 2018 at 7:47 PM, Colin McCabe <cmccabe@apache.org>
> >>>> wrote:
> >>>>
> >>>> Thanks, Sasaki.
> >>>>>
> >>>>> Colin
> >>>>>
> >>>>> On Sat, Apr 28, 2018, at 00:55, Sasaki Toru wrote:
> >>>>>
> >>>>>> Hi Colin, Jason,
> >>>>>>
> >>>>>> Thank you for your beneficial comment.
> >>>>>> I have updated my Pull Request to show git commit hash in version
> >>>>>> information.> In my current Pull Request, we cat get the
result such
> >>>>>>
> >>>>> below:
> >>>>>
> >>>>>>     $ bin/kafka-topics.sh --version
> >>>>>>     (snip)
> >>>>>>     2.0.0-SNAPSHOT (Commit:f3876cd9617faf7e)
> >>>>>>
> >>>>>>
> >>>>>> I have also updated to accept double-dash for this option (--
> >>>>>> version) only.>
> >>>>>>
> >>>>>> Many thanks,
> >>>>>> Sasaki
> >>>>>>
> >>>>>> From: Jason Gustafson <jason@confluent.io>
> >>>>>>> Date: 2018-04-25 9:42 GMT+09:00
> >>>>>>> Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's
> >>>>>>> commands> > To: dev <dev@kafka.apache.org>
> >>>>>>>
> >>>>>>>
> >>>>>>> +1 on adding the git commit id to the output. We often encounter
> >>>>>>> environments which are testing off of trunk or have modifications
> >>>>>>>
> >>>>>> on
> >>>
> >>>> top of> > an existing release.
> >>>>>>>
> >>>>>>> -Jason
> >>>>>>>
> >>>>>>> On Tue, Apr 24, 2018 at 10:06 AM, Colin McCabe <cmccabe@apache.org
> >>>>>>> wrote:> >
> >>>>>>>
> >>>>>>>> On Tue, Apr 24, 2018, at 05:36, Sasaki Toru wrote:
> >>>>>>>>
> >>>>>>>>> Hi Jason, Colin,
> >>>>>>>>>
> >>>>>>>>> Thank you for your comment, and I'm sorry for late
reply.
> >>>>>>>>>
> >>>>>>>>>    > we refactored all of the tools so that we
could use a common
> >>>>>>>>>    > set of> >>> base options,
> >>>>>>>>>    > it might be a little annoying to have to
continue supporting
> >>>>>>>>>    > both> >>> variations.
> >>>>>>>>>
> >>>>>>>>> I feel KIP-14 is good idea (I'm sorry, I didn't
know about
> >>>>>>>>> it), and> >>> I think it's no longer
necessary both single-dash
> >>>>>>>>>
> >>>>>>>> and
> >>>
> >>>> double-
> >>>>>
> >>>>>> dash are> >>> supported when this KIP will be accepted,
as you
> >>>>>>>>>
> >>>>>>>> said.
> >>>
> >>>> I revise my Pull Request to support single-dash option only.
> >>>>>>>>>
> >>>>>>>>> Some my code in kafka-run-class.sh will become unnecessary
when
> >>>>>>>>> KIP-14> >>> is accepted.
> >>>>>>>>> But I will keep this as temporary, because I seem
that it's
> >>>>>>>>>
> >>>>>>>> useful>
> >>>
> >>>> before KIP-14 accepted.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>    > Can you give a little more detail about
what would be
> >>>>>>>>>    > displayed when> >>> the version
command was used?
> >>>>>>>>>
> >>>>>>>>> As Ismael said, the version string is got from
> >>>>>>>>> AppInfoParser#getVersion.> >>>
> >>>>>>>>> In my Pull Request, we can get the result such as
below::
> >>>>>>>>>
> >>>>>>>>>      $ bin/kafka-topics.sh --version
> >>>>>>>>>      (snip)
> >>>>>>>>>      Kafka 1.2.0-SNAPSHOT
> >>>>>>>>>
> >>>>>>>> Hi Sasaki,
> >>>>>>>>
> >>>>>>>> Thanks for the info.  Can you add this to the KIP?
> >>>>>>>>
> >>>>>>>> Also, I really think we should include the git hash.
> >>>>>>>>
> >>>>>>>> best,
> >>>>>>>> Colin
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Many thanks,
> >>>>>>>>> Sasaki
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> From: Ismael Juma <ismael@juma.me.uk>
> >>>>>>>>>> Date: 2018-04-24 3:44 GMT+09:00
> >>>>>>>>>> Subject: Re: [DISCUSS] KIP-278: Add version
option to Kafka's
> >>>>>>>>>> commands> >>>> To: dev <dev@kafka.apache.org>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> FYI, the injection via the build process that
is mentioned here
> >>>>>>>>>>
> >>>>>>>>> already
> >>>>>>>
> >>>>>>>> happens. See AppInfoParser.
> >>>>>>>>>>
> >>>>>>>>>> Ismael
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Apr 23, 2018 at 9:39 AM, Colin McCabe
> >>>>>>>>>> <cmccabe@apache.org>> >> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Sasaki,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the KIP.  I think a version flag
is a good idea.
> >>>>>>>>>>>
> >>>>>>>>>>> Can you give a little more detail about
what would be
> >>>>>>>>>>>
> >>>>>>>>>> displayed
> >>
> >>> when> >> the
> >>>>>>>>>>> version command was used?
> >>>>>>>>>>>
> >>>>>>>>>>> We clearly want the version number, but
we probably also want
> >>>>>>>>>>>
> >>>>>>>>>> to
> >>>
> >>>> know> >> if
> >>>>>>>>>>> this is an official release, or a random
SNAPSHOT from a
> >>>>>>>>>>>
> >>>>>>>>>> branch.
> >>>
> >>>> If> >> this
> >>>>>>>>>>> is a release candidate, we probably want
the RC number as
> >>>>>>>>>>>
> >>>>>>>>>> well,
> >>
> >>> like> >>>>> "1.1-rc3"  We also want a git hash.  This
can be
> >>>>>>>>>>>
> >>>>>>>>>> injected by the> > build
> >>>>>
> >>>>>> process.  In the case of an official release, where the source
> >>>>>>>>>>> code> >> is not
> >>>>>>>>>>> under git, we can pull it from a file.
> >>>>>>>>>>>
> >>>>>>>>>>> For example, hadoop's version output looks
like this:
> >>>>>>>>>>>
> >>>>>>>>>>>     > cmccabe@aurora:~/Downloads/hadoop-2.8.3>
./bin/hadoop
> >>>>>>>>>>>     > version> >>>>>
   > Hadoop 2.8.3
> >>>>>>>>>>>     > Subversion
> >>>>>>>>>>>     > https://git-wip-us.apache.org/repos/asf/hadoop.git
-r>
> >>>>>>>>>>>
> >>>>>>>>>> b3fe56402d908019d99af1f1f4fc65cb1d1436a2
> >>>>>
> >>>>>>     > Compiled by jdu on 2017-12-05T03:43Z
> >>>>>>>>>>>     > Compiled with protoc 2.5.0
> >>>>>>>>>>>     > From source with checksum 9ff4856d824e983fa510d3f843e3f1
> >>>>>>>>>>>
> >>>>>>>>>> 9d>
> >>>
> >>>>     > This command was run using /home/cmccabe/Downloads/
> >>>>>>>>>>
> >>>>>>>>>>> hadoop-2.8.3/share/hadoop/common/hadoop-common-2.8.3.jar
> >>>>>>>>>>>
> >>>>>>>>>>> (The "subversion" line here is a little
weird -- it now refers
> >>>>>>>>>>> to> >> git, not
> >>>>>>>>>>> svn)
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Apr 11, 2018, at 13:58, Jason Gustafson
wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hey Sasaki,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yeah, I don't feel too strongly about
only supporting --
> >>>>>>>>>>>> version. I> >> agree
> >>>>>>>>>>>>
> >>>>>>>>>>> it
> >>>>>>>>>>>
> >>>>>>>>>>>> may help discoverability given the current
approach. On the
> >>>>>>>>>>>> other> >> hand,
> >>>>>>>>>>>>
> >>>>>>>>>>> if
> >>>>>>>>>>>
> >>>>>>>>>>>> we refactored all of the tools so that
we could use a common
> >>>>>>>>>>>> set of> >> base
> >>>>>>>>>>>> options, it might be a little annoying
to have to continue
> >>>>>>>>>>>>
> >>>>>>>>>>> supporting
> >>>>>>>
> >>>>>>>> both
> >>>>>>>>>>>
> >>>>>>>>>>>> variations. For example, tool standardization
was proposed in
> >>>>>>>>>>>> KIP-14> >> and
> >>>>>>>>>>>> I'm still holding out hope that someone
will have time to
> >>>>>>>>>>>>
> >>>>>>>>>>> pick
> >>
> >>> this> >> work
> >>>>>>>>>>>> back up. It's always easier to add an
option than remove one,
> >>>>>>>>>>>> so I'm> >>>>>>
slightly inclined to have only --version for
> >>>>>>>>>>>>
> >>>>>>>>>>> now.
> >>>
> >>>> What do you
> >>>>>
> >>>>>> think?> >>>>> The double dash version is more
consistent with
> >>>>>>>>>>>>
> >>>>>>>>>>> how
> >>>
> >>>> our other
> >>>>>
> >>>>>> flags> >> work.
> >>>>>>>>>>> In general, I feel that if --version is
supported, --help
> >>>>>>>>>>>
> >>>>>>>>>> should
> >>>
> >>>> say> >> so.
> >>>>>>>>>>> best,
> >>>>>>>>>>> Colin
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Jason
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Tue, Apr 10, 2018 at 12:00 AM, Sasaki
Toru <
> >>>>>>>>>>>>
> >>>>>>>>>>> sasakitoa@oss.nttdata.com
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Jason
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you for helpful comments.
I updated wiki based on your
> >>>>>>>>>>>>>
> >>>>>>>>>>>> advice.
> >>>>>>>
> >>>>>>>> I thought this option was relatively common and making
> >>>>>>>>>>>>> maintenance> >>>>
easy
> >>>>>>>>>>>>> was also important.
> >>>>>>>>>>>>> However, as you said, it is not
good that version option
> >>>>>>>>>>>>>
> >>>>>>>>>>>> won't
> >>>
> >>>> be> >>>>> shown up
> >>>>>>>>>>>>> in help description.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I thought accepting both single-dash
and double-dash will
> >>>>>>>>>>>>>
> >>>>>>>>>>>> help
> >>>
> >>>> to> >> find
> >>>>>>>>>>>>> this option.
> >>>>>>>>>>>>> In my approach this option won't
be showed, but most of
> >>>>>>>>>>>>> software> >> which
> >>>>>>>>>>>>>
> >>>>>>>>>>>> has
> >>>>>>>>>>>
> >>>>>>>>>>>> this option accepts either single-dash
or double-dash.
> >>>>>>>>>>>>> I guess it doesn't need to support
both if we take another
> >>>>>>>>>>>>> way.> >>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> @Ted Yeah, you're right. Sorry about
the confusion.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Since we're here, I think this
KIP is a nice improvement.
> >>>>>>>>>>>>>> It's> >>>>>
definitely
> >>>>>>>>>>>>>> nice to have an easy way to
check the version. That said,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> do
> >>
> >>> we> >>>> really
> >>>>>>>>>>>>>> need
> >>>>>>>>>>>>>> to support both `-version` and
`--version`? The latter is
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> consistent
> >>>>>>>>
> >>>>>>>>> with
> >>>>>>>>>>>
> >>>>>>>>>>>> our current tools.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Also, I think the approach we've
taken is basically to
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> build
> >>
> >>> the> >>>>> --version
> >>>>>>>>>>>>>> functionality into the bash
script. This is nice because it
> >>>>>>>>>>>>>> saves> > a
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> lot of
> >>>>>>>>>>>
> >>>>>>>>>>>> work to update the commands individually
and we don't need
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> to
> >>>
> >>>> do> >>>>> anything
> >>>>>>>>>>>>>> when we add new tools. The downside
is that `--version`
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> won't
> >>>
> >>>> show> >> up
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> as
> >>>>>>>>>>>
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>> option in any of the --help
output. Not sure if that is too
> >>>>>>>>>>>>>> big of> >> a
> >>>>>>>>>>>>>> problem, but maybe worth mentioning
this in the rejected
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> alternatives
> >>>>>>>>
> >>>>>>>>> section.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -Jason
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Apr 4, 2018 at 9:42
AM, Ted Yu <yuzhihong@gmail.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Jason:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Maybe your reply was intended
for another KIP ?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> KIP-278 is about adding
version option, not timeout.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Apr 4, 2018 at 9:36
AM, Jason Gustafson <
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> jason@confluent.io>
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Sasaki,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for the KIP.
I think the timeout controls the
> >>>>>>>>>>>>>>>> maximum> >>>>
allowed
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> time
> >>>>>>>>>>>>>>> that the consumer will block
for the next record. Maybe
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> the>
> >>>
> >>>> meaning
> >>>>>>>
> >>>>>>>> would
> >>>>>>>>>>>>>>> be clearer with the more
concise name `--timeout`? That
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> also
> >>>
> >>>> fits> >>>>> with
> >>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> old consumer which overrides
the `consumer.timeout.ms`
> >>>>>>>>>>>>>>>> property.> >>>>>>>>>>
> >>>>>>>>>>>>>>>> By the way, it seems
like the default value was
> >>>>>>>>>>>>>>>> intentionally> >
set
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> low
> >>>>>>>>>>>
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>> both the old and new consumers,
but I'm not sure of the
> >>>>>>>>>>>>>>> reason.> > We
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> could
> >>>>>>>>>>>
> >>>>>>>>>>>> leave the default as it is if we want
to be safe, but
> >>>>>>>>>>>>>>>> increasing> >>
it
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> seems
> >>>>>>>>>>>>>>> ok to me. Perhaps we could
start a little lower, though,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> say
> >>>
> >>>> 10> >>>>> seconds?
> >>>>>>>>>>>>>>> In
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> any case, we should make
it clear to the user that the
> >>>>>>>>>>>>>>>> timeout> >>
was
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> reached.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> It's surprising to see only
the incomplete reported
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> results>
> >>>
> >>>> following a
> >>>>>>>>>>
> >>>>>>>>>>> timeout.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> Jason
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Apr 4, 2018
at 4:37 AM, Sasaki Toru <
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> sasakitoa@oss.nttdata.com>
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hello everyone,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I would like to
start a discussion for KIP 278. Cloud
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> you> >
> >>>
> >>>> please
> >>>>>
> >>>>>> give
> >>>>>>>>>>>
> >>>>>>>>>>>> comments and advice ?
> >>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>>>>>>>>>>>>>> 278+-> >>>>>>>>>>>
+Add+version+option+to+Kafka%2
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 7s+commands>
> >>>>>
> >>>>>> JIRA ticket and Pull Request are bellow:
> >>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-2061>
> >>>>>>>>>>>>>>>>> <https://github.com/apache/kafka/pull/639>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Many thanks,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Sasaki
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com)
NTT DATA
> >>>>>>>>>>>>>>>>> CORPORATION>
>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com)
NTT DATA CORPORATION
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA
CORPORATION
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION
> >>>>>>
> >>>>>>
> >>>>>
> > --
> > Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION
> >
> >

Mime
View raw message