flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximilian Michels <...@apache.org>
Subject Re: Advice on [FLINK-2021]: Rework examples to use new ParameterTool
Date Thu, 10 Sep 2015 13:28:53 GMT
I think the primary concern was flink-examples but if you're on it,
you can also modify the other examples.

On Thu, Sep 10, 2015 at 12:43 PM, Behrouz Derakhshan
<behrouz.derakhshan@gmail.com> wrote:
> Hi,
>
> So my understanding was that the changes are only meant for flink-examples
> package. But each package has its own set of examples.
> And all of them has to be changed.
> Is that OK?
>
> @Ufuk: I agree, I create a ticket for adding Javadocs.
>
> BR,
> Behrouz
>
>
> On Wed, Sep 9, 2015 at 3:53 PM, Maximilian Michels <mxm@apache.org> wrote:
>
>> It would be nice to support both non-positional and positional
>> arguments. Like in
>>
>> > posarg1 posarg2 --nonpos1 nonpos1value --nonpos2 nonpos2value
>>
>> The arguments should also be named but should be expected at a fixed
>> position counting from the left ignoring non-positional arguments.
>>
>> For the time being, it would also be ok with me if we ported all
>> examples to non-positional arguments.
>>
>> On Fri, Sep 4, 2015 at 2:46 PM, Behrouz Derakhshan
>> <behrouz.derakhshan@gmail.com> wrote:
>> > Yes, I was referring mostly to blog posts and other websites and was
>> > wondering if breaking them is an issue or not.
>> > I have already created a subtask to add support for positional arguments
>> (
>> > FLINK-2621 <https://issues.apache.org/jira/browse/FLINK-2621>), so the
>> > examples would be backward compatible.
>> > The problem with that is, we have to detect from the arguments to the
>> > program, if they are positional or key/value and parse them accordingly.
>> > But if everyone is OK with completely switching to ParameterTool and
>> > breaking the support for the old way of executing the examples, then my
>> job
>> > would be also a lot easier.
>> >
>> >
>> >
>> > On Fri, Sep 4, 2015 at 2:34 PM, Robert Metzger <rmetzger@apache.org>
>> wrote:
>> >
>> >> If you are referring to this training material (
>> >>
>> >>
>> https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataArtisans/flinkTraining/exercises/dataStreamJava/rideCleansing/RideCleansing.java
>> >> ),
>> >> some of the examples are actually already using the ParameterTool.
>> >>
>> >> The problem are probably websites / blogposts etc. that show how to use
>> the
>> >> Flink examples. But I think its fine to break these. All example jars
>> >> contain the version number. If the way we pass arguments to the examples
>> >> changes between 0.9 and 0.10, that should be fine.
>> >>
>> >> I think using the ParameterTool for the examples will improve the
>> >> readability of the examples a lot. Right now, all examples have a
>> >> (copy-pasted) parseParameters() method, which is doing very simplistic
>> >> parameter parsing.
>> >>
>> >> The PT tool also allows to show the input parameters in the web
>> interface.
>> >>
>> >> So I'm voting for doing a breaking change and using parameters such as
>> >> "--input hdfs:/// --output hdfs:/// --iterations 15".
>> >>
>> >> On Fri, Sep 4, 2015 at 1:05 PM, Behrouz Derakhshan <
>> >> behrouz.derakhshan@gmail.com> wrote:
>> >>
>> >> > Will do.
>> >> >
>> >> > Thanks,
>> >> > Behrouz
>> >> >
>> >> > On Fri, Sep 4, 2015 at 11:29 AM, Maximilian Michels <mxm@apache.org>
>> >> > wrote:
>> >> >
>> >> > > Hi Behrouz,
>> >> > >
>> >> > > I would create a new sub-task under the original issue that
>> introduce
>> >> > > the ParameterTool: https://issues.apache.org/jira/browse/FLINK-1525
>> >> > >
>> >> > > Cheers,
>> >> > > Max
>> >> > >
>> >> > > On Fri, Sep 4, 2015 at 11:17 AM, Behrouz Derakhshan
>> >> > > <behrouz.derakhshan@gmail.com> wrote:
>> >> > > > Hi Max,
>> >> > > >
>> >> > > > What you said makes sense, for "ParameterTool doesn't seem
to
>> support
>> >> > > > positional arguments :) but we could fix that." should we
create a
>> >> > > separate
>> >> > > > ticket or should it also be part of FLINK-2021 ?
>> >> > > >
>> >> > > > BR,
>> >> > > > Behrouz
>> >> > > >
>> >> > > >
>> >> > > > On Fri, Sep 4, 2015 at 10:55 AM, Maximilian Michels <
>> mxm@apache.org>
>> >> > > wrote:
>> >> > > >
>> >> > > >> Hi Behrouz,
>> >> > > >>
>> >> > > >> Thanks for starting the discussion. If I understand your
question
>> >> > > >> correctly, you are asking if it breaks the training or
other
>> >> external
>> >> > > >> material if we convert the Flink examples to make use
of the
>> >> > > >> ParameterTool?
>> >> > > >>
>> >> > > >> We could make the changes such that the examples will
accept the
>> >> same
>> >> > > >> parameters but use the ParameterTool internally to verify
the
>> >> > > >> parameters and print usage information. I think most
examples
>> simply
>> >> > > >> use positional arguments and we could keep it that way.
The only
>> >> > > >> problem is that the ParameterTool doesn't seem to support
>> positional
>> >> > > >> arguments :) but we could fix that.
>> >> > > >>
>> >> > > >> Cheers,
>> >> > > >> Max
>> >> > > >>
>> >> > > >> On Thu, Sep 3, 2015 at 5:50 PM, Behrouz Derakhshan
>> >> > > >> <behrouz.derakhshan@gmail.com> wrote:
>> >> > > >> > Hi,
>> >> > > >> >
>> >> > > >> > I had at look at this ticket FLINK-2021
>> >> > > >> > <https://issues.apache.org/jira/browse/FLINK-2021>,
there
>> isn't
>> >> > much
>> >> > > to
>> >> > > >> do
>> >> > > >> > from a technical stand point and it kinda makes
sense to use
>> the
>> >> new
>> >> > > >> > "ParameterTool", since it is being used in most
of the other
>> part
>> >> of
>> >> > > the
>> >> > > >> > code base.
>> >> > > >> > The only question is do we really want to do it,
since I'm
>> >> guessing
>> >> > > some
>> >> > > >> of
>> >> > > >> > the training materials, slides and articles are
referencing
>> these
>> >> > > >> examples
>> >> > > >> > and updating those might be a burden.
>> >> > > >> >
>> >> > > >> > Let me know what you guys think, either I can start
working on
>> it
>> >> or
>> >> > > we
>> >> > > >> can
>> >> > > >> > just resolve it for good.
>> >> > > >> >
>> >> > > >> > Cheers,
>> >> > > >> > Behrouz
>> >> > > >>
>> >> > >
>> >> >
>> >>
>>

Mime
View raw message