apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pramod Immaneni <pra...@datatorrent.com>
Subject A proposal for Malhar
Date Fri, 27 May 2016 06:25:31 GMT
As you all know the continued success and growth of an open source project
is dependent on new members joining the project and contributing. This will
depend on how accessible the project is for new folks to make meaningful
contributions. For a project like ours where the code base has been in
development for years, it can be quite daunting for new members to just
pick up and make contributions. We need to find ways to make it easier for
people to do so. Malhar, namely the operator library, is an area where
people can contribute without requiring deep knowledge or expertise.

We have seen operators take time to mature as evidenced by the road taken
by some of our commonly used operators to reach production quality. This is
due to the fact that apart from the core functionality the operator is
trying to implement there are many other aspects to address such as
performance, idempotency, processing semantics and scalability. It would be
difficult even for folks familiar with all these aspects to get everything
right the first time around and produce comprehensive operators let alone
first time contributors. At the same time operators cannot reach this
maturity level unless they get used in different scenarios and get a good
look at by different people. In maturity I am also including API stability.

I would like to propose creation of a space inside Malhar, a sub-folder,
where contributions can first go in if they are not fully ready and when
they mature can be moved out of the sub-folder into an existing module or a
new module of its own, the package paths can remain the same. The
evaluation bar for contributions into this space would be more permissive
than it is today, it would require the functionality the operator was
developed for be working but will not necessitate that all fault tolerant
and scalability aspects be addressed. It will also allow new operators that
are variations of existing operators till such time as we can determine if
the new functionality can be subsumed by the original operator or it makes
sense for the new operator to exist as a separate entity. It will be up-to
committers and contributors to work together and make the decisions as to
whether the individual contributions go into this space or are ready to
just go into the regular modules.

What does everyone think.

Thanks,
Pramod

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message