apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashwin Chandra Putta <ashwinchand...@gmail.com>
Subject Re: A proposal for Malhar
Date Fri, 27 May 2016 20:05:06 GMT
Thanks for the proposal to drop the barrier. This will enable a lot of
functional contribution to the code base. We can always make a given
operator production ready as and when an opportunity arises. I would go
even further and categorize operators into the following which require
various degrees of graduation criteria for maturity.

1. Input Operators --> needs partitioning, fault tolerance, idempotency
2. Output Operators --> needs partitioning, fault tolerance, exactly once
writes behavior
3. Stateless Processing Operators --> default
4. Stateful Processing Operators --> needs partitioning, fault tolerance

Also, we should have a common goal as a community to keep maturing
operators.

Regards,
Ashwin.

On Fri, May 27, 2016 at 9:59 AM, Sasha Parfenov <sashap@apache.org> wrote:

> +1
>
> I think this is a great idea to lower the barrier for entry and encourage
> more contributions for Apex.  And checklist for each operator is very
> valuable for end-users who do not want to spend time going through every
> line of code, to provide better understanding and confidence in various
> operator aspects including processing semantics, partitioning, idempotency,
> etc.
>
> Thanks,
> Sasha
>
> On Fri, May 27, 2016 at 9:32 AM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > I think this is a good idea and the committer can help in determining
> this
> > without putting all the burden on the contributor. One way would be to
> list
> > what is missing in terms of platform capabilities and under what
> scenarios
> > the operators cannot be used or unsure whether they will work. As you
> > mentioned it could be informally done in a javadoc or we could introduce
> > special programming language constructs to denote these.
> >
> > On Fri, May 27, 2016 at 8:22 AM, Ganelin, Ilya <
> > Ilya.Ganelin@capitalone.com>
> > wrote:
> >
> > > If we're going to adopt partially completed code, I propose that every
> > new
> > > Malhar operator then contain a checklist as a comment within the class.
> > >
> > > This checklist would be defined by the community and would track the
> > > current development state of the operator. That way there are no
> > unexpected
> > > surprises.
> > >
> > >
> > >
> > > Sent with Good (www.good.com)
> > > ________________________________
> > > From: Amol Kekre <amol@datatorrent.com>
> > > Sent: Friday, May 27, 2016 10:13:06 AM
> > > To: dev@apex.apache.org
> > > Cc: dev@apex.incubator.apache.org
> > > Subject: Re: A proposal for Malhar
> > >
> > > This is a very good idea and will greatly help contributors. The
> > > requirements to submit code to this Malhar folder should be very
> > minimal. A
> > > few that come to my mind
> > >
> > > - Should compile
> > > - License of the external lib (if any) should be Apache compliant
> license
> > >  // Need to see if this is part of ASF guidelines
> > >
> > > Everything else including naming, idempotency, performance, ... should
> be
> > > waived.
> > >
> > > Thks
> > > Amol
> > >
> > >
> > > On Thu, May 26, 2016 at 11:25 PM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > As you all know the continued success and growth of an open source
> > > project
> > > > is dependent on new members joining the project and contributing.
> This
> > > will
> > > > depend on how accessible the project is for new folks to make
> > meaningful
> > > > contributions. For a project like ours where the code base has been
> in
> > > > development for years, it can be quite daunting for new members to
> just
> > > > pick up and make contributions. We need to find ways to make it
> easier
> > > for
> > > > people to do so. Malhar, namely the operator library, is an area
> where
> > > > people can contribute without requiring deep knowledge or expertise.
> > > >
> > > > We have seen operators take time to mature as evidenced by the road
> > taken
> > > > by some of our commonly used operators to reach production quality.
> > This
> > > is
> > > > due to the fact that apart from the core functionality the operator
> is
> > > > trying to implement there are many other aspects to address such as
> > > > performance, idempotency, processing semantics and scalability. It
> > would
> > > be
> > > > difficult even for folks familiar with all these aspects to get
> > > everything
> > > > right the first time around and produce comprehensive operators let
> > alone
> > > > first time contributors. At the same time operators cannot reach this
> > > > maturity level unless they get used in different scenarios and get a
> > good
> > > > look at by different people. In maturity I am also including API
> > > stability.
> > > >
> > > > I would like to propose creation of a space inside Malhar, a
> > sub-folder,
> > > > where contributions can first go in if they are not fully ready and
> > when
> > > > they mature can be moved out of the sub-folder into an existing
> module
> > > or a
> > > > new module of its own, the package paths can remain the same. The
> > > > evaluation bar for contributions into this space would be more
> > permissive
> > > > than it is today, it would require the functionality the operator was
> > > > developed for be working but will not necessitate that all fault
> > tolerant
> > > > and scalability aspects be addressed. It will also allow new
> operators
> > > that
> > > > are variations of existing operators till such time as we can
> determine
> > > if
> > > > the new functionality can be subsumed by the original operator or it
> > > makes
> > > > sense for the new operator to exist as a separate entity. It will be
> > > up-to
> > > > committers and contributors to work together and make the decisions
> as
> > to
> > > > whether the individual contributions go into this space or are ready
> to
> > > > just go into the regular modules.
> > > >
> > > > What does everyone think.
> > > >
> > > > Thanks,
> > > > Pramod
> > > >
> > > ________________________________________________________
> > >
> > > The information contained in this e-mail is confidential and/or
> > > proprietary to Capital One and/or its affiliates and may only be used
> > > solely in performance of work or services for Capital One. The
> > information
> > > transmitted herewith is intended only for use by the individual or
> entity
> > > to which it is addressed. If the reader of this message is not the
> > intended
> > > recipient, you are hereby notified that any review, retransmission,
> > > dissemination, distribution, copying or other use of, or taking of any
> > > action in reliance upon this information is strictly prohibited. If you
> > > have received this communication in error, please contact the sender
> and
> > > delete the material from your computer.
> > >
> >
>



-- 

Regards,
Ashwin.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message