kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasaki Toru <sasaki...@oss.nttdata.com>
Subject Re: [DISCUSS] KIP-278: Add version option to Kafka's commands
Date Fri, 11 May 2018 01:46:44 GMT
  I started voting thread for this KIP. Thanks, Jason and Colin.
> From: Colin McCabe <cmccabe@apache.org>
> Date: 2018-05-10 2:53 GMT+09:00
> Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's commands
> To: dev@kafka.apache.org
>
>
> +1. Thanks, Sasaki.
>
> Colin
>
> On Wed, May 9, 2018, at 09:15, Jason Gustafson wrote:
>> Hi Sasaki,
>>
>> Thanks for the update. The KIP looks good to me. I'd suggest moving to a
>> vote.
>>
>> Thanks,
>> Jason
>>
>> On Mon, May 7, 2018 at 7:08 AM, Sasaki Toru <sasakitoa@oss.nttdata.com>
>> wrote:
>>
>>> Hi Manikumar, Colin,
>>>
>>> Thank you for your comment.
>>>
>>> As Colin said, I proposed adding an option to show version information
> of
>>> "local" tool,
>>> because many software have this option and I think Kafka also needs this
>>> one.
>>>
>>> As you said, the function to show remote Kafka version is useful,
>>> but I think it is better to create new KIP because this function has
> some
>>> points which should be considered.
>>>
>>> If you have any better ideas, could you please tell us?
>>>
>>>
>>> Many thanks,
>>> Sasaki
>>>
>>> From: Manikumar <manikumar.reddy@gmail.com>
>>>> Date: 2018-05-03 4:11 GMT+09:00
>>>>
>>>> Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's commands
>>>> To: dev <dev@kafka.apache.org>
>>>>
>>>>
>>>> Hi Colin,
>>>>
>>>> Thanks for explanation. It's definitely useful to have  --version flag.
>>>>
>>>> kafka-broker-api-versions.sh gives the API versions, not Kafka release
>>>> version.
>>>> Is not easy to figure out release version from API versions. Currently
>>>> release version is available via metric/JMX.
>>>> If required, we can implement this in future.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> On Wed, May 2, 2018 at 10:58 PM, Colin McCabe <cmccabe@apache.org>
> wrote:
>>>> Hi Manikumar,
>>>>> We already have a tool for getting the Kafka broker API versions,
>>>>> "./bin/kafka-broker-api-versions.sh".  It was added as part of KIP-97.
>>>>>
>>>>> What Saski is proposing here is having a way of getting the version of
>>>>> locally installed Kafka software, which may be different from the
> server
>>>>> version.  Many pieces of software offer a --version flag, and it's
> always
>>>>> understood to refer to the local version of the software, not a
> version
>>>>> running somewhere else.  The user has no way to get this information
> now,
>>>>> other than perhaps trying to look at he names of jar files.
>>>>>
>>>>> cheers,
>>>>> Colin
>>>>>
>>>>> On Tue, May 1, 2018, at 08:20, Manikumar wrote:
>>>>>
>>>>>> I assume the intent of the KIP to find out the Kafka broker
> version.  In
>>>>>> this case, maybe we should expose
>>>>>> version using a Kafka request. This will help the remote
> scripts/tools
>>>>> to
>>>>> query the Kafka version.
>>>>>> scripts (kafka-topics.sh, kafka-configs.sh, etc..)  may run from
> remote
>>>>>> machines  and may use
>>>>>> older Kafka versions. In this case, current proposal prints on the
> older
>>>>>> version.
>>>>>>
>>>>>> On Tue, May 1, 2018 at 7:47 PM, Colin McCabe <cmccabe@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> Thanks, Sasaki.
>>>>>>> Colin
>>>>>>>
>>>>>>> On Sat, Apr 28, 2018, at 00:55, Sasaki Toru wrote:
>>>>>>>
>>>>>>>> Hi Colin, Jason,
>>>>>>>>
>>>>>>>> Thank you for your beneficial comment.
>>>>>>>> I have updated my Pull Request to show git commit hash in
version
>>>>>>>> information.> In my current Pull Request, we cat get the
result
> such
>>>>>>> below:
>>>>>>>
>>>>>>>>      $ bin/kafka-topics.sh --version
>>>>>>>>      (snip)
>>>>>>>>      2.0.0-SNAPSHOT (Commit:f3876cd9617faf7e)
>>>>>>>>
>>>>>>>>
>>>>>>>> I have also updated to accept double-dash for this option
(--
>>>>>>>> version) only.>
>>>>>>>>
>>>>>>>> Many thanks,
>>>>>>>> Sasaki
>>>>>>>>
>>>>>>>> From: Jason Gustafson <jason@confluent.io>
>>>>>>>>> Date: 2018-04-25 9:42 GMT+09:00
>>>>>>>>> Subject: Re: [DISCUSS] KIP-278: Add version option to
Kafka's
>>>>>>>>> commands> > To: dev <dev@kafka.apache.org>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> +1 on adding the git commit id to the output. We often
encounter
>>>>>>>>> environments which are testing off of trunk or have modifications
>>>>>>>>>
>>>>>>>> on
>>>>>> top of> > an existing release.
>>>>>>>>> -Jason
>>>>>>>>>
>>>>>>>>> On Tue, Apr 24, 2018 at 10:06 AM, Colin McCabe <cmccabe@apache.org
>>>>>>>>> wrote:> >
>>>>>>>>>
>>>>>>>>>> On Tue, Apr 24, 2018, at 05:36, Sasaki Toru wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Jason, Colin,
>>>>>>>>>>>
>>>>>>>>>>> Thank you for your comment, and I'm sorry for
late reply.
>>>>>>>>>>>
>>>>>>>>>>>     > we refactored all of the tools so that
we could use a
> common
>>>>>>>>>>>     > set of> >>> base options,
>>>>>>>>>>>     > it might be a little annoying to have
to continue
> supporting
>>>>>>>>>>>     > both> >>> variations.
>>>>>>>>>>>
>>>>>>>>>>> I feel KIP-14 is good idea (I'm sorry, I didn't
know about
>>>>>>>>>>> it), and> >>> I think it's no longer
necessary both single-dash
>>>>>>>>>>>
>>>>>>>>>> and
>>>>>> double-
>>>>>>>> dash are> >>> supported when this KIP will be
accepted, as you
>>>>>>>>>> said.
>>>>>> I revise my Pull Request to support single-dash option only.
>>>>>>>>>>> Some my code in kafka-run-class.sh will become
unnecessary when
>>>>>>>>>>> KIP-14> >>> is accepted.
>>>>>>>>>>> But I will keep this as temporary, because I
seem that it's
>>>>>>>>>>>
>>>>>>>>>> useful>
>>>>>> before KIP-14 accepted.
>>>>>>>>>>>     > Can you give a little more detail about
what would be
>>>>>>>>>>>     > displayed when> >>> the
version command was used?
>>>>>>>>>>>
>>>>>>>>>>> As Ismael said, the version string is got from
>>>>>>>>>>> AppInfoParser#getVersion.> >>>
>>>>>>>>>>> In my Pull Request, we can get the result such
as below::
>>>>>>>>>>>
>>>>>>>>>>>       $ bin/kafka-topics.sh --version
>>>>>>>>>>>       (snip)
>>>>>>>>>>>       Kafka 1.2.0-SNAPSHOT
>>>>>>>>>>>
>>>>>>>>>> Hi Sasaki,
>>>>>>>>>>
>>>>>>>>>> Thanks for the info.  Can you add this to the KIP?
>>>>>>>>>>
>>>>>>>>>> Also, I really think we should include the git hash.
>>>>>>>>>>
>>>>>>>>>> best,
>>>>>>>>>> Colin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Many thanks,
>>>>>>>>>>> Sasaki
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From: Ismael Juma <ismael@juma.me.uk>
>>>>>>>>>>>> Date: 2018-04-24 3:44 GMT+09:00
>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-278: Add version
option to Kafka's
>>>>>>>>>>>> commands> >>>> To: dev <dev@kafka.apache.org>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> FYI, the injection via the build process
that is mentioned here
>>>>>>>>>>>>
>>>>>>>>>>> already
>>>>>>>>>> happens. See AppInfoParser.
>>>>>>>>>>>> Ismael
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Apr 23, 2018 at 9:39 AM, Colin McCabe
>>>>>>>>>>>> <cmccabe@apache.org>> >> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Sasaki,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the KIP.  I think a version
flag is a good idea.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can you give a little more detail about
what would be
>>>>>>>>>>>>>
>>>>>>>>>>>> displayed
>>>>> when> >> the
>>>>>>>>>>>>> version command was used?
>>>>>>>>>>>>>
>>>>>>>>>>>>> We clearly want the version number, but
we probably also want
>>>>>>>>>>>>>
>>>>>>>>>>>> to
>>>>>> know> >> if
>>>>>>>>>>>>> this is an official release, or a random
SNAPSHOT from a
>>>>>>>>>>>>>
>>>>>>>>>>>> branch.
>>>>>> If> >> this
>>>>>>>>>>>>> is a release candidate, we probably want
the RC number as
>>>>>>>>>>>>>
>>>>>>>>>>>> well,
>>>>> like> >>>>> "1.1-rc3"  We also want a git hash.  This
can be
>>>>>>>>>>>> injected by the> > build
>>>>>>>> process.  In the case of an official release, where the source
>>>>>>>>>>>>> code> >> is not
>>>>>>>>>>>>> under git, we can pull it from a file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For example, hadoop's version output
looks like this:
>>>>>>>>>>>>>
>>>>>>>>>>>>>      > cmccabe@aurora:~/Downloads/hadoop-2.8.3>
./bin/hadoop
>>>>>>>>>>>>>      > version> >>>>>
   > Hadoop 2.8.3
>>>>>>>>>>>>>      > Subversion
>>>>>>>>>>>>>      > https://git-wip-us.apache.org/repos/asf/hadoop.git
-r>
>>>>>>>>>>>>>
>>>>>>>>>>>> b3fe56402d908019d99af1f1f4fc65cb1d1436a2
>>>>>>>>      > Compiled by jdu on 2017-12-05T03:43Z
>>>>>>>>>>>>>      > Compiled with protoc 2.5.0
>>>>>>>>>>>>>      > From source with checksum 9ff4856d824e983fa510d3f843e3f1
>>>>>>>>>>>>>
>>>>>>>>>>>> 9d>
>>>>>>      > This command was run using /home/cmccabe/Downloads/
>>>>>>>>>>>>> hadoop-2.8.3/share/hadoop/common/hadoop-common-2.8.3.jar
>>>>>>>>>>>>>
>>>>>>>>>>>>> (The "subversion" line here is a little
weird -- it now refers
>>>>>>>>>>>>> to> >> git, not
>>>>>>>>>>>>> svn)
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Apr 11, 2018, at 13:58, Jason
Gustafson wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey Sasaki,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yeah, I don't feel too strongly about
only supporting --
>>>>>>>>>>>>>> version. I> >> agree
>>>>>>>>>>>>>>
>>>>>>>>>>>>> it
>>>>>>>>>>>>>
>>>>>>>>>>>>>> may help discoverability given the
current approach. On the
>>>>>>>>>>>>>> other> >> hand,
>>>>>>>>>>>>>>
>>>>>>>>>>>>> if
>>>>>>>>>>>>>
>>>>>>>>>>>>>> we refactored all of the tools so
that we could use a common
>>>>>>>>>>>>>> set of> >> base
>>>>>>>>>>>>>> options, it might be a little annoying
to have to continue
>>>>>>>>>>>>>>
>>>>>>>>>>>>> supporting
>>>>>>>>>> both
>>>>>>>>>>>>>> variations. For example, tool standardization
was proposed in
>>>>>>>>>>>>>> KIP-14> >> and
>>>>>>>>>>>>>> I'm still holding out hope that someone
will have time to
>>>>>>>>>>>>>>
>>>>>>>>>>>>> pick
>>>>> this> >> work
>>>>>>>>>>>>>> back up. It's always easier to add
an option than remove one,
>>>>>>>>>>>>>> so I'm> >>>>>>
slightly inclined to have only --version for
>>>>>>>>>>>>>>
>>>>>>>>>>>>> now.
>>>>>> What do you
>>>>>>>> think?> >>>>> The double dash version is
more consistent with
>>>>>>>>>>>>> how
>>>>>> our other
>>>>>>>> flags> >> work.
>>>>>>>>>>>>> In general, I feel that if --version
is supported, --help
>>>>>>>>>>>>>
>>>>>>>>>>>> should
>>>>>> say> >> so.
>>>>>>>>>>>>> best,
>>>>>>>>>>>>> Colin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Apr 10, 2018 at 12:00 AM,
Sasaki Toru <
>>>>>>>>>>>>>>
>>>>>>>>>>>>> sasakitoa@oss.nttdata.com
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hi Jason
>>>>>>>>>>>>>>> Thank you for helpful comments.
I updated wiki based on your
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> advice.
>>>>>>>>>> I thought this option was relatively common and making
>>>>>>>>>>>>>>> maintenance> >>>>
easy
>>>>>>>>>>>>>>> was also important.
>>>>>>>>>>>>>>> However, as you said, it is not
good that version option
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> won't
>>>>>> be> >>>>> shown up
>>>>>>>>>>>>>>> in help description.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I thought accepting both single-dash
and double-dash will
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> help
>>>>>> to> >> find
>>>>>>>>>>>>>>> this option.
>>>>>>>>>>>>>>> In my approach this option won't
be showed, but most of
>>>>>>>>>>>>>>> software> >> which
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> has
>>>>>>>>>>>>>> this option accepts either single-dash
or double-dash.
>>>>>>>>>>>>>>> I guess it doesn't need to support
both if we take another
>>>>>>>>>>>>>>> way.> >>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @Ted Yeah, you're right. Sorry
about the confusion.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Since we're here, I think
this KIP is a nice improvement.
>>>>>>>>>>>>>>>> It's> >>>>>
definitely
>>>>>>>>>>>>>>>> nice to have an easy way
to check the version. That said,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> do
>>>>> we> >>>> really
>>>>>>>>>>>>>>>> need
>>>>>>>>>>>>>>>> to support both `-version`
and `--version`? The latter is
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> consistent
>>>>>>>>>>> with
>>>>>>>>>>>>>> our current tools.
>>>>>>>>>>>>>>>> Also, I think the approach
we've taken is basically to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> build
>>>>> the> >>>>> --version
>>>>>>>>>>>>>>>> functionality into the bash
script. This is nice because it
>>>>>>>>>>>>>>>> saves> > a
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> lot of
>>>>>>>>>>>>>> work to update the commands individually
and we don't need
>>>>>>>>>>>>>>> to
>>>>>> do> >>>>> anything
>>>>>>>>>>>>>>>> when we add new tools. The
downside is that `--version`
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> won't
>>>>>> show> >> up
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> an
>>>>>>>>>>>>>>>> option in any of the --help
output. Not sure if that is too
>>>>>>>>>>>>>>>> big of> >> a
>>>>>>>>>>>>>>>> problem, but maybe worth
mentioning this in the rejected
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> alternatives
>>>>>>>>>>> section.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -Jason
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Apr 4, 2018 at 9:42
AM, Ted Yu <yuzhihong@gmail.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>> Jason:
>>>>>>>>>>>>>>>>> Maybe your reply was
intended for another KIP ?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> KIP-278 is about adding
version option, not timeout.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Apr 4, 2018 at
9:36 AM, Jason Gustafson <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> jason@confluent.io>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> Hi Sasaki,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the KIP.
I think the timeout controls the
>>>>>>>>>>>>>>>>>> maximum> >>>>
allowed
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>> that the consumer will
block for the next record. Maybe
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the>
>>>>>> meaning
>>>>>>>>>> would
>>>>>>>>>>>>>>>>> be clearer with the more
concise name `--timeout`? That
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> also
>>>>>> fits> >>>>> with
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> old consumer which overrides
the `consumer.timeout.ms`
>>>>>>>>>>>>>>>>>> property.> >>>>>>>>>>
>>>>>>>>>>>>>>>>>> By the way, it seems
like the default value was
>>>>>>>>>>>>>>>>>> intentionally>
> set
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> low
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> both the old and new
consumers, but I'm not sure of the
>>>>>>>>>>>>>>>>> reason.> > We
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> could
>>>>>>>>>>>>>> leave the default as it is if we
want to be safe, but
>>>>>>>>>>>>>>>>>> increasing> >>
it
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>>>>> ok to me. Perhaps we
could start a little lower, though,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> say
>>>>>> 10> >>>>> seconds?
>>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> any case, we should make
it clear to the user that the
>>>>>>>>>>>>>>>>>> timeout> >>
was
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> reached.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's surprising to see
only the incomplete reported
>>>>>>>>>>>>>>>>> results>
>>>>>> following a
>>>>>>>>>>>>> timeout.
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Apr 4, 2018
at 4:37 AM, Sasaki Toru <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> sasakitoa@oss.nttdata.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I would like
to start a discussion for KIP 278. Cloud
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> you> >
>>>>>> please
>>>>>>>> give
>>>>>>>>>>>>>> comments and advice ?
>>>>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>>>>> 278+-> >>>>>>>>>>>
+Add+version+option+to+Kafka%2
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 7s+commands>
>>>>>>>> JIRA ticket and Pull Request are bellow:
>>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-2061>
>>>>>>>>>>>>>>>>>>> <https://github.com/apache/kafka/pull/639>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Many thanks,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Sasaki
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com)
NTT DATA
>>>>>>>>>>>>>>>>>>> CORPORATION>
>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com)
NTT DATA CORPORATION
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA
CORPORATION
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION
>>>>>>>>
>>>>>>>>
>>> --
>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION
>>>
>>>

-- 
Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION


Mime
View raw message