Mailing-List: contact general-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@incubator.apache.org
MIME-Version: 1.0
Date: Tue, 27 Oct 2015 21:52:19 -0700
Message-ID: 
 <CAGU5spcoXAh+eS0bj=vQH9jm_abJ1DzrQF_Gfn1n94_JVmmBWA@mail.gmail.com>
Subject: [VOTE] Accept SystemML into Apache Incubator
From: Luciano Resende <luckbr1975@gmail.com>
To: "general@incubator.apache.org" <general@incubator.apache.org>
Content-Type: multipart/alternative; boundary=001a114043d6941adf052322f5d9

--001a114043d6941adf052322f5d9
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

After initial discussion, please vote on the acceptance of SystemML
Project for incubation at the Apache Incubator. The full proposal is
available at the end of this message and on the wiki at :

https://wiki.apache.org/incubator/SystemML
<http://wiki.apache.org/incubator/Nuvem>

Please cast your votes:

[ ] +1, bring SystemML into Incubator
[ ] +0, I don't care either way
[ ] -1, do not bring SystemML into Incubator, because...

The vote is open for the next 72 hours and only votes from the
Incubator PMC are binding.


=3D SystemML =3D

=3D=3D Abstract =3D=3D

SystemML provides declarative large-scale machine learning (ML) that aims
at flexible specification of ML algorithms and automatic generation of
hybrid runtime plans ranging from single node, in-memory computations, to
distributed computations on Apache Hadoop MapReduce and  Apache Spark. ML
algorithms are expressed in an R-like syntax, that includes linear algebra
primitives, statistical functions, and ML-specific constructs. This
high-level language significantly increases the productivity of data
scientists as it provides (1) full flexibility in expressing custom
analytics, and (2) data independence from the underlying input formats and
physical data representations. Automatic optimization according to data
characteristics such as distribution on the disk file system, and sparsity
as well as processing characteristics in the distributed environment like
number of nodes, CPU, memory per node, ensures both efficiency and
scalability.

=3D=3D Proposal =3D=3D

The goal of SystemML is to create a commercial friendly, scalable and
extensible machine learning framework for data scientists to create or
extend machine learning algorithms using a declarative syntax. The machine
learning framework enables data scientists to develop algorithms locally
without the need of a distributed cluster, and scale up and scale out the
execution of these algorithms to distributed Apache Hadoop MapReduce or
Apache Spark clusters.

=3D=3D Background =3D=3D

SystemML started as a research project in the IBM Almaden Research Center
around 2007 aiming to enable data scientists to develop machine learning
algorithms independent of data and cluster characteristics.

=3D=3D Rationale =3D=3D

SystemML enables the specification of machine learning algorithms using a
declarative machine learning (DML) language. DML includes linear algebra
primitives, statistical functions, and additional constructs. This
high-level language significantly increases the productivity of data
scientists as it provides (1) full flexibility in expressing custom
analytics and (2) data independence from the underlying input formats and
physical data representations.

SystemML computations can be executed in a variety of different modes. It
supports single node in-memory computations and large-scale distributed
cluster computations. This allows the user to quickly prototype new
algorithms in local environments but automatically scale to large data
sizes as well without changing the algorithm implementation.

Algorithms specified in DML are dynamically compiled and optimized based on
data and cluster characteristics using rule-based and cost-based
optimization techniques. The optimizer automatically generates hybrid
runtime execution plans ranging from in-memory single-node execution to
distributed computations on Apache Spark or Apache Hadoop MapReduce. This
ensures both efficiency and scalability. Automatic optimization reduces or
eliminates the need to hand-tune distributed runtime execution plans and
system configurations.

=3D=3D Initial Goals =3D=3D

The initial goals to move SystemML to the Apache Incubator is to broaden
the community foster the contributions from data scientists to develop new
machine learning algorithms and enhance the existing ones. Ultimately, this
may lead to the creation of an industry standard in specifying machine
learning algorithms.

=3D=3D Current Status =3D=3D

The initial code has been developed at the IBM Almaden Research Center in
California and has recently been made available in GitHub under the Apache
Software License 2.0. The project currently supports a single node (in
memory computation) as well as distributed computations utilizing Apache
Hadoop MapReduce or Apache Spark clusters.

=3D=3D=3D Meritocracy =3D=3D=3D

We plan to invest in supporting a meritocracy. We will discuss the
requirements in an open forum. Several companies have already expressed
interest in this project, and we intend to invite additional developers to
participate. We will encourage and monitor community participation so that
privileges can be extended to those that contribute operating to the
standard of meritocracy that Apache emphasizes.

=3D=3D=3D Community =3D=3D=3D

The need for a generic scalable and declarative machine learning approach
in the open source is tremendous, so there is a potential for a very large
community. We believe that SystemML=E2=80=99s extensible architecture, decl=
arative
syntax, cost based optimizer and its alignment with Spark will further
encourage community participation not only in enhancing the infrastructure
but also speed up the creation of algorithms for a wide range of use
cases.  We expect that over time SystemML will attract a large community.

=3D=3D=3D Alignment =3D=3D=3D

The initial committers strongly believe that a generic scalable and
declarative machine learning approach for machine learning will gain
broader adoption as an open source, community driven project, where the
community can contribute not only to the core components, but also to a
growing collection of algorithms which will leverage the optimizations and
ease of scaling in SystemML. Our hope is that the Apache Spark, Apache
Hadoop and other communities will find tremendous value in SystemML and
this will foster further collaboration between these projects furthering
the already existing integration points.

=3D=3D Known Risks =3D=3D

To-date, development has been sponsored by IBM and coordinated mostly by
the core team of researchers at the IBM Almaden Research Center.

For SystemML to fully transition to an "Apache Way" governance model, it
needs to start embracing the meritocracy-centric way of growing the
community of contributors.

=3D=3D=3D Orphaned Products =3D=3D=3D

The SystemML developers and previous sponsor have a long-term interest in
use and maintenance of the code and there is also hope that growing a
diverse community around the project will become a guarantee against the
project becoming orphaned. We feel that it is also important to put formal
governance in place both for the project and the contributors as the
project expands. We feel ASF is the best location for this.

=3D=3D=3D Inexperience with Open Source =3D=3D=3D

The current SystemML set of contributors are very diverse regarding
participation in Open Source. While some initial members are experiencing
an open source project for the first time, others have been contributing
and mentoring various Apache and non-Apache open source projects.

=3D=3D=3D Reliance on Salaried Developers =3D=3D=3D

SystemML currently receives substantial support from salaried developers.
However, they are all passionate about the project, and we are confident
that the project will continue even if no salaried developers contribute to
the project. We are committed to recruiting additional committers including
non-salaried developers.


=3D=3D=3D Relationships with Other Apache Products =3D=3D=3D

Currently, SystemML integrates with Apache Hadoop MapReduce and Apache
Spark as underlying computational distributed runtimes.

=3D=3D=3D An Excessive Fascination with the Apache Brand =3D=3D=3D

SystemML solves a real need for generic scalable and declarative machine
learning approach for machine learning in the Apache Hadoop and Spark
ecosystems, something that has been addressed in a very ad hoc manner so
far by multiple Apache projects. Our rationale for developing SystemML as
an Apache project is detailed in the Rationale section. We believe that the
Apache brand and community process will help us attract more contributors
to this project, and help establish ubiquitous APIs.


=3D=3D Documentation =3D=3D

Documentation regarding SystemML is available in the current GitHub
repository https://github.com/SparkTC/systemml/tree/master/system-ml/docs.

=3D=3D Initial Source =3D=3D

Initial source is available on GitHub under the Apache License 2.0

https://github.com/SparkTC/systemml

=3D=3D Source and Intellectual Property Submission Plan =3D=3D

We know of no legal encumbrances in the transfer of source code and rights
to Apache. In fact, given the internal IBM due diligence performed on the
source code during open sourcing, we expect the code base to be free from
any IP issues.

=3D=3D External Dependencies =3D=3D

SystemML is written in Java and currently supports Apache Hadoop MapReduce
and Apache Spark runtimes.

To the best of our knowledge, all dependencies of SystemML are distributed
under Apache compatible licenses. Upon acceptance to the incubator, we
would begin a thorough analysis of all transitive dependencies to verify
this fact and introduce license checking into the build and release process
(for instance integrating Apache Rat).

Cryptography
N/A

=3D=3D Required Resources =3D=3D

=3D=3D=3D Mailing lists =3D=3D=3D
      * private@sysml.incubator.apache.org (moderated subscriptions)
      * commits@sysml.incubator.apache.org
      * dev@sysml.incubator.apache.org

=3D=3D=3D Git Repository =3D=3D=3D
      * https://git-wip-us.apache.org/repos/asf/incubator-sysml.git

=3D=3D=3D Issue Tracking =3D=3D=3D
      * JIRA (SYSML)

=3D=3D Initial Committers =3D=3D

 * Luciano Resende (lresende AT apache DOT org)
 * Berthold Reinwald (reinwald AT us DOT ibm DOT com)
 * Matthias Boehm (mboehm AT us DOT ibm DOT com)
 * Shirish Tatikonda (statiko AT us DOT ibm DOT com)
 * Niketan Pansare (npansar AT us DOT ibm DOT com)
 * Prithviraj Sen (senp AT us DOT ibm DOT com)
 * Alexandre V Evfimievski (evfimi AT us DOT ibm DOT com)
 * Fred Reiss (frreiss AT us DOT ibm DOT com)
 * Deron Eriksson (deron AT us DOT ibm DOT com)
 * Arvind Surve (asurve AT us DOT ibm DOT com)
 * Mike Dusenberry (mwdusenb AT us DOT ibm DOT com)
 * Reynold Xin   (rxin AT apache DOT org)
 * Xiangrui Meng (meng AT apache DOT org)
 * Joseph Bradley (jkbradley AT apache DOT org)
 * Patrick Wendell (pwendell AT apache DOT org)
 * Holden Karau (holden AT apache DOT org)
 * DB Tsai (dbtsai AT apache DOT org)

=3D=3D Affiliations =3D=3D

 * DataBricks: Reynold Xin, Xiangrui Meng, Joseph Bradley, Patrick Wendell
 * Netflix: DB Tsai
 * IBM: Luciano Resende, Berthold Reinwald, Matthias Boehm, Shirish
Tatikonda, Niketan Pansare, Prithviraj Sen, Alexandre V Evfimievski, Fred
Reiss, Deron Eriksson, Arvind Surve, Mike Dusenberry and Holden Karau.

=3D=3D Sponsors =3D=3D

=3D=3D=3D Champion =3D=3D=3D
 * Luciano Resende

=3D=3D=3D Nominated Mentors =3D=3D=3D
 * Luciano Resende
 * Reynold Xin
 * Patrick Wendell
 * Rich Bowen

=3D=3D=3D Sponsoring Entity =3D=3D=3D
We would like to propose the Apache Incubator to sponsor this project.


--=20
Luciano Resende
http://people.apache.org/~lresende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

--001a114043d6941adf052322f5d9--