incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yaniv Rodenski <roa...@gmail.com>
Subject Re: [DISCUSS] Apache Amaterasu Incubator Proposal
Date Tue, 28 Mar 2017 20:32:44 GMT
Absolutely,

I'll start the discussion in our slack (no mailing list yet :))

Cheers,
Yaniv

On Tue, 28 Mar 2017 at 11:43 pm, Jean-Baptiste Onofré <jb@nanthrax.net>
wrote:

> Thanks for the update. Let see with the team about the name.
>
> By the way, we are looking for a third mentor if someone is interested.
>
> Regards
> JB
>
> On 03/27/2017 10:17 PM, Ross Gardler wrote:
> > Exciting stuff, it may have already been said but the name is pretty
> bad. To my (native) English ear it sounds like "Amateur".
> >
> > Ross
> >
> > -----Original Message-----
> > From: Jean-Baptiste Onofré [mailto:jb@nanthrax.net]
> > Sent: Monday, March 20, 2017 11:39 PM
> > To: general@incubator.apache.org
> > Subject: Re: [DISCUSS] Apache Amaterasu Incubator Proposal
> >
> > Hi all,
> >
> > gently reminder about this thread.
> >
> > I would like to start a formal vote pretty soon.
> >
> > Thanks,
> > Regards
> > JB
> >
> > On 03/07/2017 09:49 PM, Jean-Baptiste Onofré wrote:
> >> Hi all,
> >>
> >> I would like to submit a new proposal to bring Amaterasu to the Apache
> >> Software Foundation incubator.
> >>
> >> The proposal is included below and available on the wiki:
> >>
> >> https://wiki.apache.org/incubator/AmaterasuProposal
> >>
> >> We are eager to get your comments and questions.
> >>
> >> Thanks !
> >> JB (on behalf of the Amaterasu community)
> >>
> >> = Apache Amaterasu =
> >>
> >> == Abstract ==
> >>
> >> Apache Amaterasu is a framework providing continuous deployment for
> >> Big Data pipelines.
> >>
> >> It provides the following capabilities:
> >>
> >>  * '''Continuous integration''' tools to '''package pipelines and run
> tests'''.
> >>  * A repository to store those packaged applications: the
> >> '''applications repository'''.
> >>  * A repository to store the pipelines, and engine configuration (for
> >> instance, location of the Spark master, etc.): per environment - the
> >> '''configuration repository'''.
> >>  * A '''dashboard''' to monitor the pipelines.
> >>  * A '''DSL and integration hooks''' allowing third parties to easily
> integrate.
> >>
> >> == Proposal ==
> >>
> >> Amaterasu is a simple and powerful framework to build and dispense
> pipelines.
> >> It aims to help data engineers and data scientists to compose,
> >> configure, test, package, deploy and execute data pipelines written
> >> using multiple tools, languages and frameworks. Amaterasu provides a
> >> standard repo structure to package big data pipelines, a YAML based
> >> Domain Specific Languages (DSL) for data engineers, data scientists
> >> and operations engineers to manage complex pipelines throughout their
> entire lifecycle (Dev, UAT, Prod, etc.).
> >>
> >> == Background ==
> >>
> >> Amaterasu is a relatively new project that was created to deal with
> >> some of the issues that as Consultants, we have seen recurring at
> different client sites.
> >> Mainly the need to continuously deploy complex pipelines built in
> >> multiple tools and languages.
> >> Amaterasu started as a pet project and is currently being evaluated by
> >> a couple of organizations, supported by the contributors, on a
> >> personal time and voluntary bases.
> >>
> >> == Rational ==
> >>
> >> As software engineers working on big data projects we have straggled
> >> for a long time to apply the same CI/CD practices that have become the
> >> standard in the software industry for the last few years. While some
> >> of them are possible, for example Apache Spark is easy to unit test.
> >> However large scale pipelines are more complex and often use data,
> >> which might be un-structured as integration point, which requires heavy
> integration tests.
> >>
> >> To automate such tests and complex deployments, we have found the need
> >> to often handcraft scripts and use a mixture tools, so we have decided
> >> to finally build a tool we can apply in a general way and not on a
> project by project basis.
> >>
> >> Another issue Amaterasu is trying to tackle is the Integrating between
> >> the work of software engineers, data scientists, and sometimes
> >> operations engineers. The approach Amaterasu takes to integrate
> >> between those three schools of thought it to provide a simple YAML
> >> based DSL that provides a simple way to integrate different pipeline
> >> written in the native tools for each task (R, Spark in different
> languages, etc.).
> >>
> >> == Initial Goals ==
> >>
> >> Our initial goals are to bring Amaterasu into the ASF, transition
> >> internal engineering processes into the open, and foster a
> >> collaborative development model according to the "Apache Way".
> >>
> >> In addition, we intend to continue the development of Amaterasu, add
> >> new features as well as  integrate better with other frameworks,
> including:
> >>
> >>  * Apache Arrow
> >>  * Apache Hive
> >>  * Apache Drill
> >>  * Apache Beam
> >>  * Apache YARN
> >>  * Farther and more complete integration with Apache Spark
> >>
> >> Other frameworks will be evaluated after those initial goals are
> reached.
> >>
> >> == Current Status ==
> >>
> >> Amaterasu is preview state but provide a large set of features. We
> >> plan to stabilize and head to a first production ready release during
> >> the incubation process. The current license is already Apache 2.0.
> >>
> >> === Meritocracy ===
> >>
> >> We intend to radically expand the initial developer and user community
> >> by running the project in accordance with the "Apache Way". Users and
> >> new contributors will be treated with respect and welcomed. By
> >> participating in the community and providing quality patches/support
> >> that move the project forward, they will earn merit. They also will be
> >> encouraged to provide non-code contributions (documentation, events,
> >> community management, etc.) and will gain merit for doing so. Those
> >> with a proven support and quality track record will be encouraged to
> become committers.
> >>
> >> === Community ===
> >>
> >> As a relatively new project, Amaterasu has a small, but growing
> community.
> >> Amaterasu is an open project, not just with it’s source code but also
> >> with our discussions which are held openly in our slack
> >> https://shintoio.slack.com which contains channels for design, tech
> and future directions discussions.
> >>
> >> If Amaterasu is accepted for incubation, the primary initial goal is
> >> to build a large and strong community. We are confident that Amaterasu
> >> can become a key project for big data operations, which hopefully will
> >> create a large community of users and developers.
> >>
> >> === Known Risks ===
> >>
> >> Development has been sponsored mostly by a one company. For the
> >> project to fully transition to the Apache Way governance model,
> >> development must shift towards the meritocracy-centric model of
> >> growing a community of contributors balanced with the needs for extreme
> stability and core implementation coherency.
> >>
> >> === Orphaned products ===
> >>
> >> We are fully committed on Amaterasu. A few organizations have
> >> expressed their interest in using Amaterasu.
> >>
> >> === Inexperience with Open Source ===
> >>
> >> We have been developing and using open source software for a long time.
> >> Additionally, several ASF veterans have agreed to mentor the project
> >> and they are listed in this proposal. The project will rely on their
> >> guidance and collective wisdom to quickly transition the entire team
> >> of initial committers towards practicing the Apache Way.
> >>
> >> === Reliance on Salaried Developers ===
> >>
> >> Most of the current contributors are employed in the Big Data space.
> >> While they might wander from their current employers, they are
> >> unlikely to venture far from their core expertises and thus will
> >> continue to be engaged with the project regardless of their current
> employers.
> >>
> >> === An Excessive Fascination with the Apache Brand ===
> >>
> >> While we intend to leverage the Apache ‘branding’ when talking to
> >> other projects as testament of our project’s ‘neutrality’, we have no
> >> plans for making use of Apache brand in press releases nor posting
> >> billboards advertising acceptance of Amaterasu into Apache Incubator.
> >>
> >> The main purpose in applying for Apache incubation is due to the fact
> >> that Amaterasu is built with integration already in mind for many
> >> tools which are Apache projects, and we see Amaterasu as an extension
> >> of these projects. We hope that by being an Apache project, we can
> >> integrate better, and collaborate more effectively with the relevant
> >> projects. As Amaterasu matures, we see mutual benefits for all involved.
> >>
> >> === Initial Source ===
> >>
> >> https://github.com/shintoio/amaterasu
> >>
> >> === External Dependencies ===
> >>
> >> All external dependencies are licensed under an Apache 2.0 license or
> >> Apache-compatible license. As we grow the Amaterasu community we will
> >> configure our build process to require and validate all contributions
> >> and dependencies are licensed under the Apache 2.0 license or are under
> an Apache-compatible license.
> >>
> >>  * Apache Spark
> >>  * Apache Hadoop
> >>  * Apache Maven (maven-core)
> >>  * Apache Commons
> >>  * Apache Log4j
> >>  * Apache Mesos
> >>  * Apache Zookeeper
> >>  * Apache Curator
> >>  * Scala
> >>  * Junit
> >>  * Py4j
> >>
> >> Future versions are planned to integrate with:
> >>
> >>  * Apache YARN
> >>  * Apache Hive
> >>  * Apache Drill
> >>
> >> === Required Resources ===
> >>
> >> ==== Mailing lists ====
> >>
> >>  * private@amaterasu.incubator.apache.org (moderated subscriptions)
> >>  * commits@amaterasu.incubator.apache.org
> >>  * dev@amaterasu.incubator.apache.org
> >>  * issues@amaterasu.incubator.apache.org
> >>
> >> ==== Git Repository ====
> >>
> >>  * https://git-wip-us.apache.org/repos/asf/incubator-amaterasu.git
> >>
> >> ==== Issue Tracking ====
> >>
> >>  * JIRA Project Amaterasu
> >>
> >> ==== Initial Committers =====
> >>
> >>  * Yaniv Rodenski
> >>  * Jean-Baptiste Onofré
> >>  * Eyal Ben Ivri
> >>  * Karel Alfonso
> >>  * Kirupagaran (Kirupa) Devarajan
> >>  * Nadav Har Tzvi
> >>
> >> ==== Affiliations ====
> >>
> >>  * Yaniv Rodenski - Shinto
> >>  * Jean-Baptiste Onofré - Talend
> >>  * Olivier Lamy - Webtide
> >>
> >> ==== Sponsors ====
> >>
> >> ==== Champion ====
> >>
> >>  * Jean-Baptiste Onofré
> >>
> >> ==== Mentors ====
> >>
> >>  * Jean-Baptiste Onofré
> >>  * Olivier Lamy
> >>
> >> ==== Sponsoring Entity ====
> >>
> >> The Apache Incubator
> >>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message