apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject Re: A proposal for Malhar
Date Fri, 27 May 2016 17:52:47 GMT
On Fri, May 27, 2016 at 10:29 AM, Chinmay Kolhatkar <chinmay@datatorrent.com
> wrote:

> +1 for the idea of incubating space in malhar. This will not just ensure
> more contributions but a faster development of operators for putting into
> firsthand use.
>
> Though, I believe we need to build a process around that, right from the
> point where operator developer starts to develop till the operator matures
> and goes into some stable space in malhar. But the process should be
> lightweight enough for it not to become a new reason for lesser
> contributions.
>

Agreed, we should call out the steps and have well documented guidelines
for these. As Thomas pointed out earlier, contributions guidelines document
is a good place to put these.


>
> Here are some thoughts around process (contributing guidelines), I think we
> need to take care of for this:
> 1. What class address spaces the operators space should go in. This is to
> ensure that there are lesser backward compatibility issues exist after
> moving to stable space. In the best scenario, it should be the change of
> maven dependency and everything else should work fine.

2. As Ilya pointed out, min checklist criteria is required for both getting
> into incubating space & then into stable space. This will ensure nothing is
> missed and the quality is maintained.
> 3. IMO its very important to decide when, what, why and who of moving the
> code from incubating to stable space. This can be a grey area and process
> should define that upfront. Last thing we want to do in have some code in
> incubating but not moved to stable space for ages.
> 4. We should also decide about the cases where some serious changes are
> required in stable code, should we move it back to incubating and go
> through the process again?
> 5. Maybe we can go for Commit-Then-Review model for this incubating space.
> Its important to keep in mind that we're enabling faster development at the
> cost of probably more work in longer run. Hence suggesting CTR model
> instead of RTC model.
> 6. We should also make it clear that though this is an incubating space,
> there should not be any lax on unit testing of the operators.
>

I agree with most of what you have said and some of these should be
included in the guidelines. I would say we need further discussions on the
actual mechanisms for 3. and 4. and these could start immediately after the
current proposal, if acceptable to the group, is put in place. Regd 5, I
think we should stick with the RTC model for now and evaluate how it is
working before considering the alternative.

Thanks


> Thanks,
> Chinmay.
>
>
> On Fri, May 27, 2016 at 10:24 AM, Pramod Immaneni <pramod@datatorrent.com>
> wrote:
>
> > My comment inline
> >
> > On Fri, May 27, 2016 at 9:00 AM, David Yan <david@datatorrent.com>
> wrote:
> >
> > > This is an important change because not only will it help contributors
> > who
> > > want to contribute to Apex Malhar, it also helps users on deciding
> which
> > > operators they can use.
> > >
> >
> > Thanks
> >
> >
> > >
> > > Malhar in its current state, has way too many operators that fall in
> the
> > > "non-production quality" category. We should make it obvious to users
> > that
> > > which operators are up to par, and which operators are not, and maybe
> > even
> > > remove those that are likely not ever used in a real use case.
> > >
> >
> > I am ambivalent about revisiting older operators and doing this exercise
> as
> > this can cause unnecessary tensions. My original intent is for
> > contributions going forward.
> >
> > Thanks
> >
> >
> > > David
> > >
> > > On Fri, May 27, 2016 at 7:13 AM, Amol Kekre <amol@datatorrent.com>
> > wrote:
> > >
> > > > This is a very good idea and will greatly help contributors. The
> > > > requirements to submit code to this Malhar folder should be very
> > > minimal. A
> > > > few that come to my mind
> > > >
> > > > - Should compile
> > > > - License of the external lib (if any) should be Apache compliant
> > license
> > > >  // Need to see if this is part of ASF guidelines
> > > >
> > > > Everything else including naming, idempotency, performance, ...
> should
> > be
> > > > waived.
> > > >
> > > > Thks
> > > > Amol
> > > >
> > > >
> > > > On Thu, May 26, 2016 at 11:25 PM, Pramod Immaneni <
> > > pramod@datatorrent.com>
> > > > wrote:
> > > >
> > > > > As you all know the continued success and growth of an open source
> > > > project
> > > > > is dependent on new members joining the project and contributing.
> > This
> > > > will
> > > > > depend on how accessible the project is for new folks to make
> > > meaningful
> > > > > contributions. For a project like ours where the code base has been
> > in
> > > > > development for years, it can be quite daunting for new members to
> > just
> > > > > pick up and make contributions. We need to find ways to make it
> > easier
> > > > for
> > > > > people to do so. Malhar, namely the operator library, is an area
> > where
> > > > > people can contribute without requiring deep knowledge or
> expertise.
> > > > >
> > > > > We have seen operators take time to mature as evidenced by the road
> > > taken
> > > > > by some of our commonly used operators to reach production quality.
> > > This
> > > > is
> > > > > due to the fact that apart from the core functionality the operator
> > is
> > > > > trying to implement there are many other aspects to address such
as
> > > > > performance, idempotency, processing semantics and scalability. It
> > > would
> > > > be
> > > > > difficult even for folks familiar with all these aspects to get
> > > > everything
> > > > > right the first time around and produce comprehensive operators let
> > > alone
> > > > > first time contributors. At the same time operators cannot reach
> this
> > > > > maturity level unless they get used in different scenarios and get
> a
> > > good
> > > > > look at by different people. In maturity I am also including API
> > > > stability.
> > > > >
> > > > > I would like to propose creation of a space inside Malhar, a
> > > sub-folder,
> > > > > where contributions can first go in if they are not fully ready and
> > > when
> > > > > they mature can be moved out of the sub-folder into an existing
> > module
> > > > or a
> > > > > new module of its own, the package paths can remain the same. The
> > > > > evaluation bar for contributions into this space would be more
> > > permissive
> > > > > than it is today, it would require the functionality the operator
> was
> > > > > developed for be working but will not necessitate that all fault
> > > tolerant
> > > > > and scalability aspects be addressed. It will also allow new
> > operators
> > > > that
> > > > > are variations of existing operators till such time as we can
> > determine
> > > > if
> > > > > the new functionality can be subsumed by the original operator or
> it
> > > > makes
> > > > > sense for the new operator to exist as a separate entity. It will
> be
> > > > up-to
> > > > > committers and contributors to work together and make the decisions
> > as
> > > to
> > > > > whether the individual contributions go into this space or are
> ready
> > to
> > > > > just go into the regular modules.
> > > > >
> > > > > What does everyone think.
> > > > >
> > > > > Thanks,
> > > > > Pramod
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message