kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasaki Toru <sasaki...@oss.nttdata.com>
Subject Re: [DISCUSS] KIP-278: Add version option to Kafka's commands
Date Mon, 07 May 2018 14:08:30 GMT
Hi Manikumar, Colin,

Thank you for your comment.

As Colin said, I proposed adding an option to show version information of "local" tool,
because many software have this option and I think Kafka also needs this one.

As you said, the function to show remote Kafka version is useful,
but I think it is better to create new KIP because this function has some points which should
be considered.

If you have any better ideas, could you please tell us?


Many thanks,
Sasaki

> From: Manikumar <manikumar.reddy@gmail.com>
> Date: 2018-05-03 4:11 GMT+09:00
> Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's commands
> To: dev <dev@kafka.apache.org>
>
>
> Hi Colin,
>
> Thanks for explanation. It's definitely useful to have  --version flag.
>
> kafka-broker-api-versions.sh gives the API versions, not Kafka release
> version.
> Is not easy to figure out release version from API versions. Currently
> release version is available via metric/JMX.
> If required, we can implement this in future.
>
>
> Thanks,
>
> On Wed, May 2, 2018 at 10:58 PM, Colin McCabe <cmccabe@apache.org> wrote:
>
>> Hi Manikumar,
>>
>> We already have a tool for getting the Kafka broker API versions,
>> "./bin/kafka-broker-api-versions.sh".  It was added as part of KIP-97.
>>
>> What Saski is proposing here is having a way of getting the version of
>> locally installed Kafka software, which may be different from the server
>> version.  Many pieces of software offer a --version flag, and it's always
>> understood to refer to the local version of the software, not a version
>> running somewhere else.  The user has no way to get this information now,
>> other than perhaps trying to look at he names of jar files.
>>
>> cheers,
>> Colin
>>
>> On Tue, May 1, 2018, at 08:20, Manikumar wrote:
>>> I assume the intent of the KIP to find out the Kafka broker version.  In
>>> this case, maybe we should expose
>>> version using a Kafka request. This will help the remote scripts/tools
> to
>>> query the Kafka version.
>>> scripts (kafka-topics.sh, kafka-configs.sh, etc..)  may run from remote
>>> machines  and may use
>>> older Kafka versions. In this case, current proposal prints on the older
>>> version.
>>>
>>> On Tue, May 1, 2018 at 7:47 PM, Colin McCabe <cmccabe@apache.org> wrote:
>>>
>>>> Thanks, Sasaki.
>>>>
>>>> Colin
>>>>
>>>> On Sat, Apr 28, 2018, at 00:55, Sasaki Toru wrote:
>>>>> Hi Colin, Jason,
>>>>>
>>>>> Thank you for your beneficial comment.
>>>>> I have updated my Pull Request to show git commit hash in version
>>>>> information.> In my current Pull Request, we cat get the result such
>>>> below:
>>>>>     $ bin/kafka-topics.sh --version
>>>>>     (snip)
>>>>>     2.0.0-SNAPSHOT (Commit:f3876cd9617faf7e)
>>>>>
>>>>>
>>>>> I have also updated to accept double-dash for this option (--
>>>>> version) only.>
>>>>>
>>>>> Many thanks,
>>>>> Sasaki
>>>>>
>>>>>> From: Jason Gustafson <jason@confluent.io>
>>>>>> Date: 2018-04-25 9:42 GMT+09:00
>>>>>> Subject: Re: [DISCUSS] KIP-278: Add version option to Kafka's
>>>>>> commands> > To: dev <dev@kafka.apache.org>
>>>>>>
>>>>>>
>>>>>> +1 on adding the git commit id to the output. We often encounter
>>>>>> environments which are testing off of trunk or have modifications
>> on
>>>>>> top of> > an existing release.
>>>>>>
>>>>>> -Jason
>>>>>>
>>>>>> On Tue, Apr 24, 2018 at 10:06 AM, Colin McCabe <cmccabe@apache.org
>>>>>> wrote:> >
>>>>>>> On Tue, Apr 24, 2018, at 05:36, Sasaki Toru wrote:
>>>>>>>> Hi Jason, Colin,
>>>>>>>>
>>>>>>>> Thank you for your comment, and I'm sorry for late reply.
>>>>>>>>
>>>>>>>>    > we refactored all of the tools so that we could use
a common
>>>>>>>>    > set of> >>> base options,
>>>>>>>>    > it might be a little annoying to have to continue
supporting
>>>>>>>>    > both> >>> variations.
>>>>>>>>
>>>>>>>> I feel KIP-14 is good idea (I'm sorry, I didn't know about
>>>>>>>> it), and> >>> I think it's no longer necessary
both single-dash
>> and
>>>> double-
>>>>>>>> dash are> >>> supported when this KIP will be
accepted, as you
>> said.
>>>>>>>> I revise my Pull Request to support single-dash option only.
>>>>>>>>
>>>>>>>> Some my code in kafka-run-class.sh will become unnecessary
when
>>>>>>>> KIP-14> >>> is accepted.
>>>>>>>> But I will keep this as temporary, because I seem that it's
>> useful>
>>>>>>> before KIP-14 accepted.
>>>>>>>>
>>>>>>>>    > Can you give a little more detail about what would
be
>>>>>>>>    > displayed when> >>> the version command
was used?
>>>>>>>>
>>>>>>>> As Ismael said, the version string is got from
>>>>>>>> AppInfoParser#getVersion.> >>>
>>>>>>>> In my Pull Request, we can get the result such as below::
>>>>>>>>
>>>>>>>>      $ bin/kafka-topics.sh --version
>>>>>>>>      (snip)
>>>>>>>>      Kafka 1.2.0-SNAPSHOT
>>>>>>> Hi Sasaki,
>>>>>>>
>>>>>>> Thanks for the info.  Can you add this to the KIP?
>>>>>>>
>>>>>>> Also, I really think we should include the git hash.
>>>>>>>
>>>>>>> best,
>>>>>>> Colin
>>>>>>>
>>>>>>>
>>>>>>>> Many thanks,
>>>>>>>> Sasaki
>>>>>>>>
>>>>>>>>
>>>>>>>>> From: Ismael Juma <ismael@juma.me.uk>
>>>>>>>>> Date: 2018-04-24 3:44 GMT+09:00
>>>>>>>>> Subject: Re: [DISCUSS] KIP-278: Add version option to
Kafka's
>>>>>>>>> commands> >>>> To: dev <dev@kafka.apache.org>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> FYI, the injection via the build process that is mentioned
here
>>>>>> already
>>>>>>>>> happens. See AppInfoParser.
>>>>>>>>>
>>>>>>>>> Ismael
>>>>>>>>>
>>>>>>>>> On Mon, Apr 23, 2018 at 9:39 AM, Colin McCabe
>>>>>>>>> <cmccabe@apache.org>> >> wrote:
>>>>>>>>>> Hi Sasaki,
>>>>>>>>>>
>>>>>>>>>> Thanks for the KIP.  I think a version flag is a
good idea.
>>>>>>>>>>
>>>>>>>>>> Can you give a little more detail about what would
be
> displayed
>>>>>>>>>> when> >> the
>>>>>>>>>> version command was used?
>>>>>>>>>>
>>>>>>>>>> We clearly want the version number, but we probably
also want
>> to
>>>>>>>>>> know> >> if
>>>>>>>>>> this is an official release, or a random SNAPSHOT
from a
>> branch.
>>>>>>>>>> If> >> this
>>>>>>>>>> is a release candidate, we probably want the RC number
as
> well,
>>>>>>>>>> like> >>>>> "1.1-rc3"  We also
want a git hash.  This can be
>>>> injected by the> > build
>>>>>>>>>> process.  In the case of an official release, where
the source
>>>>>>>>>> code> >> is not
>>>>>>>>>> under git, we can pull it from a file.
>>>>>>>>>>
>>>>>>>>>> For example, hadoop's version output looks like this:
>>>>>>>>>>
>>>>>>>>>>     > cmccabe@aurora:~/Downloads/hadoop-2.8.3>
./bin/hadoop
>>>>>>>>>>     > version> >>>>>    >
Hadoop 2.8.3
>>>>>>>>>>     > Subversion
>>>>>>>>>>     > https://git-wip-us.apache.org/repos/asf/hadoop.git
-r>
>>>> b3fe56402d908019d99af1f1f4fc65cb1d1436a2
>>>>>>>>>>     > Compiled by jdu on 2017-12-05T03:43Z
>>>>>>>>>>     > Compiled with protoc 2.5.0
>>>>>>>>>>     > From source with checksum 9ff4856d824e983fa510d3f843e3f1
>> 9d>
>>>>>>>>>     > This command was run using /home/cmccabe/Downloads/
>>>>>>>>>> hadoop-2.8.3/share/hadoop/common/hadoop-common-2.8.3.jar
>>>>>>>>>>
>>>>>>>>>> (The "subversion" line here is a little weird --
it now refers
>>>>>>>>>> to> >> git, not
>>>>>>>>>> svn)
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 11, 2018, at 13:58, Jason Gustafson wrote:
>>>>>>>>>>> Hey Sasaki,
>>>>>>>>>>>
>>>>>>>>>>> Yeah, I don't feel too strongly about only supporting
--
>>>>>>>>>>> version. I> >> agree
>>>>>>>>>> it
>>>>>>>>>>> may help discoverability given the current approach.
On the
>>>>>>>>>>> other> >> hand,
>>>>>>>>>> if
>>>>>>>>>>> we refactored all of the tools so that we could
use a common
>>>>>>>>>>> set of> >> base
>>>>>>>>>>> options, it might be a little annoying to have
to continue
>>>>>> supporting
>>>>>>>>>> both
>>>>>>>>>>> variations. For example, tool standardization
was proposed in
>>>>>>>>>>> KIP-14> >> and
>>>>>>>>>>> I'm still holding out hope that someone will
have time to
> pick
>>>>>>>>>>> this> >> work
>>>>>>>>>>> back up. It's always easier to add an option
than remove one,
>>>>>>>>>>> so I'm> >>>>>> slightly
inclined to have only --version for
>> now.
>>>> What do you
>>>>>>>>>>> think?> >>>>> The double dash
version is more consistent with
>> how
>>>> our other
>>>>>>>>>> flags> >> work.
>>>>>>>>>> In general, I feel that if --version is supported,
--help
>> should
>>>>>>>>>> say> >> so.
>>>>>>>>>> best,
>>>>>>>>>> Colin
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Jason
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 10, 2018 at 12:00 AM, Sasaki Toru
<
>>>>>>> sasakitoa@oss.nttdata.com
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Jason
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you for helpful comments. I updated
wiki based on your
>>>>>> advice.
>>>>>>>>>>>> I thought this option was relatively common
and making
>>>>>>>>>>>> maintenance> >>>> easy
>>>>>>>>>>>> was also important.
>>>>>>>>>>>> However, as you said, it is not good that
version option
>> won't
>>>>>>>>>>>> be> >>>>> shown up
>>>>>>>>>>>> in help description.
>>>>>>>>>>>>
>>>>>>>>>>>> I thought accepting both single-dash and
double-dash will
>> help
>>>>>>>>>>>> to> >> find
>>>>>>>>>>>> this option.
>>>>>>>>>>>> In my approach this option won't be showed,
but most of
>>>>>>>>>>>> software> >> which
>>>>>>>>>> has
>>>>>>>>>>>> this option accepts either single-dash or
double-dash.
>>>>>>>>>>>> I guess it doesn't need to support both if
we take another
>>>>>>>>>>>> way.> >>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> @Ted Yeah, you're right. Sorry about the
confusion.
>>>>>>>>>>>>> Since we're here, I think this KIP is
a nice improvement.
>>>>>>>>>>>>> It's> >>>>> definitely
>>>>>>>>>>>>> nice to have an easy way to check the
version. That said,
> do
>>>>>>>>>>>>> we> >>>> really
>>>>>>>>>>>>> need
>>>>>>>>>>>>> to support both `-version` and `--version`?
The latter is
>>>>>>> consistent
>>>>>>>>>> with
>>>>>>>>>>>>> our current tools.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, I think the approach we've taken
is basically to
> build
>>>>>>>>>>>>> the> >>>>> --version
>>>>>>>>>>>>> functionality into the bash script. This
is nice because it
>>>>>>>>>>>>> saves> > a
>>>>>>>>>> lot of
>>>>>>>>>>>>> work to update the commands individually
and we don't need
>> to
>>>>>>>>>>>>> do> >>>>> anything
>>>>>>>>>>>>> when we add new tools. The downside is
that `--version`
>> won't
>>>>>>>>>>>>> show> >> up
>>>>>>>>>> as
>>>>>>>>>>>>> an
>>>>>>>>>>>>> option in any of the --help output. Not
sure if that is too
>>>>>>>>>>>>> big of> >> a
>>>>>>>>>>>>> problem, but maybe worth mentioning this
in the rejected
>>>>>>> alternatives
>>>>>>>>>>>>> section.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -Jason
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Apr 4, 2018 at 9:42 AM, Ted Yu
<yuzhihong@gmail.com
>>>>>> wrote:
>>>>>>>>>>>>> Jason:
>>>>>>>>>>>>>> Maybe your reply was intended for
another KIP ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> KIP-278 is about adding version option,
not timeout.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Apr 4, 2018 at 9:36 AM, Jason
Gustafson <
>>>>>>> jason@confluent.io>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Sasaki,
>>>>>>>>>>>>>>> Thanks for the KIP. I think the
timeout controls the
>>>>>>>>>>>>>>> maximum> >>>>
allowed
>>>>>>>>>>>>>> time
>>>>>>>>>>>>>> that the consumer will block for
the next record. Maybe
>> the>
>>>>>> meaning
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>> be clearer with the more concise
name `--timeout`? That
>> also
>>>>>>>>>>>>>> fits> >>>>> with
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> old consumer which overrides
the `consumer.timeout.ms`
>>>>>>>>>>>>>>> property.> >>>>>>>>>>
>>>>>>>>>>>>>>> By the way, it seems like the
default value was
>>>>>>>>>>>>>>> intentionally> > set
>>>>>>>>>> low
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> both the old and new consumers, but
I'm not sure of the
>>>>>>>>>>>>>> reason.> > We
>>>>>>>>>> could
>>>>>>>>>>>>>>> leave the default as it is if
we want to be safe, but
>>>>>>>>>>>>>>> increasing> >> it
>>>>>>>>>>>>>> seems
>>>>>>>>>>>>>> ok to me. Perhaps we could start
a little lower, though,
>> say
>>>>>>>>>>>>>> 10> >>>>> seconds?
>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> any case, we should make it clear
to the user that the
>>>>>>>>>>>>>>> timeout> >> was
>>>>>>>>>>>>>> reached.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's surprising to see only the
incomplete reported
>> results>
>>>>>>>>> following a
>>>>>>>>>>>>>>> timeout.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Jason
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Apr 4, 2018 at 4:37 AM,
Sasaki Toru <
>>>>>>>>>> sasakitoa@oss.nttdata.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>> I would like to start a discussion
for KIP 278. Cloud
>> you> >
>>>> please
>>>>>>>>>> give
>>>>>>>>>>>>>>>> comments and advice ?
>>>>>>>>>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>>>>>>>> 278+-> >>>>>>>>>>>
+Add+version+option+to+Kafka%2
>>>> 7s+commands>
>>>>>>>>>>>>>>>> JIRA ticket and Pull Request
are bellow:
>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/KAFKA-2061>
>>>>>>>>>>>>>>>> <https://github.com/apache/kafka/pull/639>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Many thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sasaki
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com)
NTT DATA
>>>>>>>>>>>>>>>> CORPORATION> >>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT
DATA CORPORATION
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>> --
>>>>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION
>>>>>>>>
>>>>> --
>>>>> Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION
>>>>>
>>>>

-- 
Sasaki Toru(sasakitoa@oss.nttdata.com) NTT DATA CORPORATION


Mime
View raw message