apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jaikit Jilka <jji...@leadferret.com>
Subject Re: A proposal for Malhar
Date Thu, 28 Jul 2016 18:37:10 GMT
Hello,

I think there is some problem while reading the properties file. I tried to add these same
properties manually before launching the application in the console. It ran but Input operator
gave java.sql.SQLFeatureNotSupportedException. But when I again tried running the application
without adding the properties manually in the console it failed. What might be the reason
for such behavior?

Thank You,

Jaikit Jilka

----- Original Message -----
From: "Lakshmi Velineni" <lakshmi@datatorrent.com>
To: "users" <users@apex.apache.org>
Sent: Thursday, July 28, 2016 11:03:00 AM
Subject: Re: A proposal for Malhar

Hi,

I created a shared google sheet and tracked the various details of
operators. Currently, the sheet contains information about operators under
lib/algo only. Link is
https://docs.google.com/a/datatorrent.com/spreadsheets/d/15IjMa-vYK6Wru4kZnUGIgg6sQAptDJ_CaWpXt3GDccM/edit?usp=sharing
.
Will update the sheet soon with lib/math too.

Thanks
Lakshmi Prasanna

On Tue, Jul 12, 2016 at 2:36 PM, David Yan <david@datatorrent.com> wrote:

> Hi Lakshmi,
>
> Thanks for volunteering.
>
> I think Pramod's suggestion of putting the operators into 3 buckets and
> Siyuan's suggestion of starting a shared Google Sheet that tracks
> individual operators are both good, with the exception that lib/streamquery
> is one unit and we probably do not need to look at individual operators
> under it.
>
> If we don't have any objection in the community, let's start the process.
>
> David
>
> On Tue, Jul 12, 2016 at 2:24 PM, Lakshmi Velineni <lakshmi@datatorrent.com
> > wrote:
>
>> I am interested to work on this.
>>
>> Regards,
>> Lakshmi prasanna
>>
>> On Tue, Jul 12, 2016 at 1:55 PM, hsy541@gmail.com <hsy541@gmail.com>
>> wrote:
>>
>> > Why not have a shared google sheet with a list of operators and options
>> > that we want to do with it.
>> > I think it's case by case.
>> > But retire unused or obsolete operators is important and we should do it
>> > sooner rather than later.
>> >
>> > Regards,
>> > Siyuan
>> >
>> > On Tue, Jul 12, 2016 at 1:09 PM, Amol Kekre <amol@datatorrent.com>
>> wrote:
>> >
>> >>
>> >> My vote is to do 2&3
>> >>
>> >> Thks
>> >> Amol
>> >>
>> >>
>> >> On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh <
>> >> VKottapalli@directv.com> wrote:
>> >>
>> >>> +1 for deprecating the packages listed below.
>> >>>
>> >>> -----Original Message-----
>> >>> From: hsy541@gmail.com [mailto:hsy541@gmail.com]
>> >>> Sent: Tuesday, July 12, 2016 12:01 PM
>> >>>
>> >>> +1
>> >>>
>> >>> On Tue, Jul 12, 2016 at 11:53 AM, David Yan <david@datatorrent.com>
>> >>> wrote:
>> >>>
>> >>> > Hi all,
>> >>> >
>> >>> > I would like to renew the discussion of retiring operators in
>> Malhar.
>> >>> >
>> >>> > As stated before, the reason why we would like to retire operators
>> in
>> >>> > Malhar is because some of them were written a long time ago before
>> >>> > Apache incubation, and they do not pertain to real use cases, are
>> not
>> >>> > up to par in code quality, have no potential for improvement, and
>> >>> > probably completely unused by anybody.
>> >>> >
>> >>> > We do not want contributors to use them as a model of their
>> >>> > contribution, or users to use them thinking they are of quality,
and
>> >>> then hit a wall.
>> >>> > Both scenarios are not beneficial to the reputation of Apex.
>> >>> >
>> >>> > The initial 3 packages that we would like to target are *lib/algo*,
>> >>> > *lib/math*, and *lib/streamquery*.
>> >>>
>> >>> >
>> >>> > I'm adding this thread to the users list. Please speak up if you
are
>> >>> > using any operator in these 3 packages. We would like to hear from
>> you.
>> >>> >
>> >>> > These are the options I can think of for retiring those operators:
>> >>> >
>> >>> > 1) Completely remove them from the malhar repository.
>> >>> > 2) Move them from malhar-library into a separate artifact called
>> >>> > malhar-misc
>> >>> > 3) Mark them deprecated and add to their javadoc that they are
no
>> >>> > longer supported
>> >>> >
>> >>> > Note that 2 and 3 are not mutually exclusive. Any thoughts?
>> >>> >
>> >>> > David
>> >>> >
>> >>> > On Tue, Jun 7, 2016 at 2:27 PM, Pramod Immaneni
>> >>> > <pramod@datatorrent.com>
>> >>> > wrote:
>> >>> >
>> >>> >> I wanted to close the loop on this discussion. In general everyone
>> >>> >> seemed to be favorable to this idea with no serious objections.
>> Folks
>> >>> >> had good suggestions like documenting capabilities of operators,
>> come
>> >>> >> up well defined criteria for graduation of operators and what
those
>> >>> >> criteria may be and what to do with existing operators that
may not
>> >>> >> yet be mature or unused.
>> >>> >>
>> >>> >> I am going to summarize the key points that resulted from the
>> >>> >> discussion and would like to proceed with them.
>> >>> >>
>> >>> >>    - Operators that do not yet provide the key platform
>> capabilities
>> >>> to
>> >>> >>    make an operator useful across different applications such
as
>> >>> >> reusability,
>> >>> >>    partitioning static or dynamic, idempotency, exactly once
will
>> >>> still be
>> >>> >>    accepted as long as they are functionally correct, have
unit
>> tests
>> >>> >> and will
>> >>> >>    go into a separate module.
>> >>> >>    - Contrib module was suggested as a place where new
>> contributions
>> >>> go in
>> >>> >>    that don't yet have all the platform capabilities and are
not
>> yet
>> >>> >> mature.
>> >>> >>    If there are no other suggestions we will go with this one.
>> >>> >>    - It was suggested the operators documentation list those
>> platform
>> >>> >>    capabilities it currently provides from the list above.
I will
>> >>> >> document a
>> >>> >>    structure for this in the contribution guidelines.
>> >>> >>    - Folks wanted to know what would be the criteria to graduate
an
>> >>> >>    operator to the big leagues :). I will kick-off a separate
>> thread
>> >>> >> for it as
>> >>> >>    I think it requires its own discussion and hopefully we
can come
>> >>> >> up with a
>> >>> >>    set of guidelines for it.
>> >>> >>    - David brought up state of some of the existing operators
and
>> >>> their
>> >>> >>    retirement and the layout of operators in Malhar in general
and
>> >>> how it
>> >>> >>    causes problems with development. I will ask him to lead
the
>> >>> >> discussion on
>> >>> >>    that.
>> >>> >>
>> >>> >> Thanks
>> >>> >>
>> >>> >> On Fri, May 27, 2016 at 7:47 PM, David Yan <david@datatorrent.com>
>> >>> wrote:
>> >>> >>
>> >>> >> > The two ideas are not conflicting, but rather complementing.
>> >>> >> >
>> >>> >> > On the contrary, putting a new process for people trying
to
>> >>> >> > contribute while NOT addressing the old unused subpar
operators
>> in
>> >>> >> > the repository
>> >>> >> is
>> >>> >> > what is conflicting.
>> >>> >> >
>> >>> >> > Keep in mind that when people try to contribute, they
always look
>> >>> >> > at the existing operators already in the repository as
examples
>> and
>> >>> >> > likely a
>> >>> >> model
>> >>> >> > for their new operators.
>> >>> >> >
>> >>> >> > David
>> >>> >> >
>> >>> >> >
>> >>> >> > On Fri, May 27, 2016 at 4:05 PM, Amol Kekre <
>> amol@datatorrent.com>
>> >>> >> wrote:
>> >>> >> >
>> >>> >> > > Yes there are two conflicting threads now. The original
thread
>> >>> >> > > was to
>> >>> >> > open
>> >>> >> > > up a way for contributors to submit code in a dir
(contrib?) as
>> >>> >> > > long
>> >>> >> as
>> >>> >> > > license part of taken care of.
>> >>> >> > >
>> >>> >> > > On the thread of removing non-used operators ->
How do we know
>> >>> >> > > what is being used?
>> >>> >> > >
>> >>> >> > > Thks,
>> >>> >> > > Amol
>> >>> >> > >
>> >>> >> > >
>> >>> >> > > On Fri, May 27, 2016 at 3:40 PM, Sandesh Hegde <
>> >>> >> sandesh@datatorrent.com>
>> >>> >> > > wrote:
>> >>> >> > >
>> >>> >> > > > +1 for removing the not-used operators.
>> >>> >> > > >
>> >>> >> > > > So we are creating a process for operator writers
who don't
>> >>> >> > > > want to understand the platform, yet wants to
contribute? How
>> >>> >> > > > big is that
>> >>> >> set?
>> >>> >> > > > If we tell the app-user, here is the code which
has not
>> passed
>> >>> >> > > > all
>> >>> >> the
>> >>> >> > > > checklist, will they be ready to use that in
production?
>> >>> >> > > >
>> >>> >> > > > This thread has 2 conflicting forces, reduce
the operators
>> and
>> >>> >> > > > make
>> >>> >> it
>> >>> >> > > easy
>> >>> >> > > > to add more operators.
>> >>> >> > > >
>> >>> >> > > >
>> >>> >> > > >
>> >>> >> > > > On Fri, May 27, 2016 at 3:03 PM Pramod Immaneni
<
>> >>> >> > pramod@datatorrent.com>
>> >>> >> > > > wrote:
>> >>> >> > > >
>> >>> >> > > > > On Fri, May 27, 2016 at 2:30 PM, Gaurav
Gupta <
>> >>> >> > > gaurav.gopi123@gmail.com>
>> >>> >> > > > > wrote:
>> >>> >> > > > >
>> >>> >> > > > > > Pramod,
>> >>> >> > > > > >
>> >>> >> > > > > > By that logic I would say let's put
all partitionable
>> >>> >> > > > > > operators
>> >>> >> > into
>> >>> >> > > > one
>> >>> >> > > > > > folder, non-partitionable operators
in another and so
>> on...
>> >>> >> > > > > >
>> >>> >> > > > >
>> >>> >> > > > > Remember the original goal of making it
easier for new
>> >>> >> > > > > members to contribute and managing those
contributions to
>> >>> >> > > > > maturity. It is
>> >>> >> not a
>> >>> >> > > > > functional level separation.
>> >>> >> > > > >
>> >>> >> > > > >
>> >>> >> > > > > > When I look at hadoop code I see these
annotations being
>> >>> >> > > > > > used at
>> >>> >> > > class
>> >>> >> > > > > > level and not at package/folder level.
>> >>> >> > > > >
>> >>> >> > > > >
>> >>> >> > > > > I had a typo in my email, I meant to say
"think of this
>> like
>> >>> >> > > > > a
>> >>> >> > > folder..."
>> >>> >> > > > > as an analogy and not literally.
>> >>> >> > > > >
>> >>> >> > > > > Thanks
>> >>> >> > > > >
>> >>> >> > > > >
>> >>> >> > > > > > Thanks
>> >>> >> > > > > >
>> >>> >> > > > > > On Fri, May 27, 2016 at 2:10 PM, Pramod
Immaneni <
>> >>> >> > > > pramod@datatorrent.com
>> >>> >> > > > > >
>> >>> >> > > > > > wrote:
>> >>> >> > > > > >
>> >>> >> > > > > > > On Fri, May 27, 2016 at 1:05
PM, Gaurav Gupta <
>> >>> >> > > > > gaurav.gopi123@gmail.com>
>> >>> >> > > > > > > wrote:
>> >>> >> > > > > > >
>> >>> >> > > > > > > > Can same goal not be achieved
by using
>> >>> >> > > org.apache.hadoop.classification.InterfaceStability.Evolving
>> >>> >> > > > /
>> >>> >> > > > > > > >
>> org.apache.hadoop.classification.InterfaceStability.Uns
>> >>> >> > > > > > > > table
>> >>> >> > > > > > annotation?
>> >>> >> > > > > > > >
>> >>> >> > > > > > >
>> >>> >> > > > > > > I think it is important to localize
the additions in
>> one
>> >>> >> place so
>> >>> >> > > > that
>> >>> >> > > > > it
>> >>> >> > > > > > > becomes clearer to users about
the maturity level of
>> >>> >> > > > > > > these,
>> >>> >> > easier
>> >>> >> > > > for
>> >>> >> > > > > > > developers to track them towards
the path to maturity
>> and
>> >>> >> > > > > > > also
>> >>> >> > > > > provides a
>> >>> >> > > > > > > clearer directive for committers
and contributors on
>> >>> >> acceptance
>> >>> >> > of
>> >>> >> > > > new
>> >>> >> > > > > > > submissions. Relying on the annotations
alone makes
>> them
>> >>> >> spread
>> >>> >> > all
>> >>> >> > > > > over
>> >>> >> > > > > > > the place and adds an additional
layer of difficulty in
>> >>> >> > > > identification
>> >>> >> > > > > > not
>> >>> >> > > > > > > just for users but also for developers
who want to find
>> >>> >> > > > > > > such
>> >>> >> > > > operators
>> >>> >> > > > > > and
>> >>> >> > > > > > > improve them. This of this like
a folder level
>> annotation
>> >>> >> where
>> >>> >> > > > > > everything
>> >>> >> > > > > > > under this folder is unstable
or evolving.
>> >>> >> > > > > > >
>> >>> >> > > > > > > Thanks
>> >>> >> > > > > > >
>> >>> >> > > > > > >
>> >>> >> > > > > > > >
>> >>> >> > > > > > > > On Fri, May 27, 2016 at
12:35 PM, David Yan <
>> >>> >> > > david@datatorrent.com
>> >>> >> > > > >
>> >>> >> > > > > > > wrote:
>> >>> >> > > > > > > >
>> >>> >> > > > > > > > > >
>> >>> >> > > > > > > > > > >
>> >>> >> > > > > > > > > > > >
>> >>> >> > > > > > > > > > > > Malhar
in its current state, has way too many
>> >>> >> operators
>> >>> >> > > > that
>> >>> >> > > > > > fall
>> >>> >> > > > > > > > in
>> >>> >> > > > > > > > > > the
>> >>> >> > > > > > > > > > > > "non-production
quality" category. We should
>> >>> >> > > > > > > > > > > > make
it
>> >>> >> > > > obvious
>> >>> >> > > > > to
>> >>> >> > > > > > > > users
>> >>> >> > > > > > > > > > > that
>> >>> >> > > > > > > > > > > > which
operators are up to par, and which
>> >>> >> > > > > > > > > > > > operators
>> >>> >> are
>> >>> >> > > not,
>> >>> >> > > > > and
>> >>> >> > > > > > > > maybe
>> >>> >> > > > > > > > > > > even
>> >>> >> > > > > > > > > > > > remove
those that are likely not ever used
>> in a
>> >>> >> > > > > > > > > > > > real
>> >>> >> > use
>> >>> >> > > > > case.
>> >>> >> > > > > > > > > > > >
>> >>> >> > > > > > > > > > >
>> >>> >> > > > > > > > > > > I am ambivalent
about revisiting older
>> operators
>> >>> >> > > > > > > > > > > and
>> >>> >> > doing
>> >>> >> > > > this
>> >>> >> > > > > > > > > exercise
>> >>> >> > > > > > > > > > as
>> >>> >> > > > > > > > > > > this can
cause unnecessary tensions. My
>> original
>> >>> >> intent
>> >>> >> > is
>> >>> >> > > > for
>> >>> >> > > > > > > > > > > contributions
going forward.
>> >>> >> > > > > > > > > > >
>> >>> >> > > > > > > > > > >
>> >>> >> > > > > > > > > > IMO it is important
to address this as well.
>> >>> >> > > > > > > > > > Operators
>> >>> >> > > outside
>> >>> >> > > > > the
>> >>> >> > > > > > > play
>> >>> >> > > > > > > > > > area should be
of well known quality.
>> >>> >> > > > > > > > > >
>> >>> >> > > > > > > > > >
>> >>> >> > > > > > > > > I think this is important,
and I don't anticipate
>> >>> >> > > > > > > > > much
>> >>> >> > tension
>> >>> >> > > if
>> >>> >> > > > > we
>> >>> >> > > > > > > > > establish clear criteria.
>> >>> >> > > > > > > > > It's not helpful if
we let the old subpar operators
>> >>> >> > > > > > > > > stay
>> >>> >> and
>> >>> >> > > put
>> >>> >> > > > up
>> >>> >> > > > > > the
>> >>> >> > > > > > > > > bars for new operators.
>> >>> >> > > > > > > > >
>> >>> >> > > > > > > > > David
>> >>> >> > > > > > > > >
>> >>> >> > > > > > > >
>> >>> >> > > > > > >
>> >>> >> > > > > >
>> >>> >> > > > >
>> >>> >> > > >
>> >>> >> > >
>> >>> >> >
>> >>> >>
>> >>> >
>> >>> >
>> >>>
>> >>
>> >>
>> >
>>
>
>

Mime
View raw message