incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <t...@cloudera.com>
Subject Re: [VOTE] Oozie to join the Incubator
Date Thu, 30 Jun 2011 05:20:07 GMT
+1 (non-binding)

On Wed, Jun 29, 2011 at 10:18 PM, Ashish <paliwalashish@gmail.com> wrote:

> +1 (non-binding)
>
> On Thu, Jun 30, 2011 at 12:40 AM, Mohammad Islam <mislam77@yahoo.com>
> wrote:
>
> > Hi All,
> >
> > The discussion about Oozie proposal is settling down. Therefore I would
> > like to
> > initiate a vote to accept Oozie as an Apache Incubator project.
> >
> > The latest proposal is pasted at the end and it could be found in the
> wiki
> > as
> > well:
> >
> > http://wiki.apache.org/incubator/OozieProposal
> >
> >
> > The related discussion thread is at:
> > http://www.mail-archive.com/general@incubator.apache.org/msg29633.html
> >
> >
> > Please cast your votes:
> >
> > [  ] +1 Accept Oozie for incubation
> > [  ] +0 Indifferent to Oozie incubation
> > [  ] -1 Reject Oozie for incubation
> >
> > This vote will close 72 hours  from now.
> >
> > Regards,
> > Mohammad
> >
> >
> > Abstract
> > Oozie is a server-based workflow scheduling and coordination system to
> > manage
> > data processing jobs for Apache HadoopTM.
> >
> > Proposal
> > Oozie is an  extensible, scalable and reliable system to define, manage,
> > schedule,  and execute complex Hadoop workloads via web services. More
> > specifically, this includes:
> >
> >        * XML-based declarative framework to specify a job or a complex
> > workflow of
> > dependent jobs.
> >
> >        * Support different types of job such as Hadoop Map-Reduce, Pipe,
> > Streaming,
> > Pig, Hive and custom java applications.
> >
> >        * Workflow scheduling based on frequency and/or data availability.
> >        * Monitoring capability, automatic retry and failure handing of
> > jobs.
> >        * Extensible and pluggable architecture to allow arbitrary grid
> > programming
> > paradigms.
> >
> >        * Authentication, authorization, and capacity-aware load
> throttling
> > to allow
> > multi-tenant software as a service.
> >
> > Background
> > Most data  processing applications require multiple jobs to achieve their
> > goals,
> > with inherent dependencies among the jobs. A dependency could be
> >  sequential,
> > where one job can only start after another job has finished.  Or it could
> > be
> > conditional, where the execution of a job depends on the  return value or
> > status
> > of another job. In other cases, parallel  execution of multiple jobs may
> be
> > permitted – or desired – to exploit  the massive pool of compute nodes
> > provided
> > by Hadoop.
> >
> > These  job dependencies are often expressed as a Directed Acyclic Graph,
> > also
> > called a workflow. A node in the workflow is typically a job (a
> >  computation on
> > the grid) or another type of action such as an eMail  notification.
> > Computations
> > can be expressed in map/reduce, Pig, Hive or  any other programming
> > paradigm
> > available on the grid. Edges of the graph  represent transitions from one
> > node
> > to the next, as the execution of a  workflow proceeds.
> >
> > Describing  a workflow in a declarative way has the advantage of
> decoupling
> > job
> > dependencies and execution control from application logic. Furthermore,
> >  the
> > workflow is modularized into jobs that can be reused within the same
> >  workflow
> > or across different workflows. Execution of the workflow is  then driven
> by
> > a
> > runtime system without understanding the application  logic of the jobs.
> > This
> > runtime system specializes in reliable and  predictable execution: It can
> > retry
> > actions that have failed or invoke a  cleanup action after termination of
> > the
> > workflow; it can monitor  progress, success, or failure of a workflow,
> and
> > send
> > appropriate alerts  to an administrator. The application developer is
> > relieved
> > from  implementing these generic procedures.
> >
> > Furthermore,  some applications or workflows need to run in periodic
> > intervals
> > or  when dependent data is available. For example, a workflow could be
> >  executed
> > every day as soon as output data from the previous 24 instances  of
> > another,
> > hourly workflow is available. The workflow coordinator  provides such
> > scheduling
> > features, along with prioritization, load  balancing and throttling to
> > optimize
> > utilization of resources in the  cluster. This makes it easier to
> maintain,
> > control, and coordinate  complex data applications.
> >
> > Nearly  three years ago, a team of Yahoo! developers addressed these
> > critical
> > requirements for Hadoop-based data processing systems by developing a
>  new
> > workflow management and scheduling system called Oozie. While it was
> >  initially
> > developed as a Yahoo!-internal project, it was designed and  implemented
> > with
> > the intention of open-sourcing. Oozie was released as a GitHub project in
> > early
> > 2010. Oozie is used in production within Yahoo and  since it has been
> > open-sourced it has been gaining adoption with  external developers
> >
> > Rationale
> > Commonly,  applications that run on Hadoop require multiple Hadoop jobs
> in
> > order
> > to  obtain the desired results. Furthermore, these Hadoop jobs are
> commonly
> >  a
> > combination of Java map-reduce jobs, Streaming map-reduce jobs, Pipes
> > map-reduce jobs, Pig jobs, Hive jobs, HDFS operations, Java programs  and
> > shell
> > scripts.
> >
> > Because  of this, developers find themselves writing ad-hoc glue programs
> > to
> > combine these Hadoop jobs. These ad-hoc programs are difficult to
> >  schedule,
> > manage, monitor and recover.
> >
> > Workflow  management and scheduling is an essential feature for
> large-scale
> > data
> > processing applications. Such applications could write the customized
> >  solution
> > that would require separate development, operational, and  maintenance
> > overhead.
> > Since it is a prevalent use-case for data  processing, the application
> > developer
> > would surely prefer a generalized  solution with little or no such
> > overhead.
> > Oozie addresses the challenge  by providing an execution framework to
> > flexibly
> > specify the job  dependency, data dependency, and time dependency. In
> > addition,
> > Oozie  provides a multi-tenant-based centralized service and the
> > opportunity to
> > optimize load and utilization while respecting SLAs.
> >
> > Oozie is built on Apache HadoopTM to schedule jobs related to various
> > Apache
> > projects such as Hadoop,  Pig, and Hive. As an Apache Open source
> project,
> > Oozie
> > is expected to  attract the larger and more diversified community that
> > currently
> > uses  such Apache sponsored projects. Additionally, users of the Hadoop
> > ecosystem can influence Oozie’s roadmap, and contribute to it. Likewise,
> >  Oozie,
> > as part of the Apache Hadoop TMecosystem, will be a great benefit to the
> > current
> > Hadoop/Pig/Hive/HBase/HCatalog community.
> >
> > Current Status
> > Meritocracy
> > Oozie  currently is a github-based open sourced project where developers
> > from
> > multiple companies are contributing to the project. Our intent with this
> > incubator proposal is to further extend this diverse developer  community
> > around
> > Oozie following the Apache meritocracy model. We plan  to continue to
> > provide
> > adequate support to new developers and to quickly  recruit those who make
> > solid
> > contributions to committer status. In  addition, Oozie will expect,
> accept,
> > and
> > work to attract contributions  from amateurs as well.
> >
> > Community
> > While an  efficient workflow management and scheduling system is critical
> > for
> > large companies with huge data processing in multi-tenant clusters, it
>  is
> > equally necessary for any non-trivial deployment. Different companies
>  are
> > currently using Oozie as a workflow scheduler for Hadoop-based data
> >  processing.
> > At Yahoo! it is being used extensively in production  clusters to process
> > thousand of jobs. Like the Oozie user community, the  Oozie developer
> > community
> > is also very strong. Developers from Yahoo!  provided the initial code
> > base, and
> > they are still the most active  contributors. In late 2010, developers
> from
> > Cloudera also started  contributing, and currently other companies (e.g.,
> > IBM)
> > are beginning to  participate.
> >
> > We currently use JIRA for issue tracking, github for code hosting and
> > Yahoo!
> > Groups for developer and user communications.
> >
> > Core Developers
> > Oozie is  currently being designed and developed by four engineers from
> > Yahoo! –
> > Mohammad Islam, Angelo Huang, Mayank Bansal, and Andreas Neumann. In
> >  addition,
> > many outside contributors are actively contributing in design  and
> > development.
> > Among them, Alejandro Abdelnur from Cloudera and Chao  Wang from IBM are
> > very
> > important contributors. All of these core  developers have deep expertise
> > in
> > Hadoop and the Hadoop Ecosystem in  general.
> >
> > Alignment
> > The ASF is a  natural host for Oozie given that it is already the home of
> > Hadoop,  Pig, Hive, and other emerging cloud software projects. Oozie was
> > designed to support Hadoop from the beginning in order to solve data
> >  processing
> > challenges in Hadoop clusters. Oozie complements the existing  Apache
> cloud
> > computing projects by providing a flexible framework for  managing
> complex
> > data
> > processing tasks.
> >
> > Known Risks
> > Orphaned Products
> > The core  developers plan to work full time on the project. There is very
> > little
> > risk of Oozie getting orphaned since large companies like Yahoo! are
> > extensively using it on their production Hadoop clusters. For example,
> >  there
> > are nearly 400 Yahoo! internal Oozie users and thousands of jobs  are
> > processed
> > hourly through Oozie in production. In addition, there are  nearly 400
> > active
> > users (including Yahoo! internal and external) in the  email community
> > where
> > nearly 15 emails are exchanged per day.  Furthermore, there were more
> than
> > 1500
> > downloads of the Oozie binary in  the last eight months from the github
> > site and
> > a large number of  downloads were conducted by other companies such as
> > Cloudera.
> > Oozie has  three major releases and more than 15 patch releases in the
> last
> > couple  of years which further demonstrates Oozie as a very active
> project.
> > We
> > plan to extend and diversify this community further through Apache.
> >
> > Inexperience with Open Source
> > The core  developers are all active users and followers of open source.
> > They are
> > already committers and contributors to the Oozie Github project. In
> >  addition,
> > they are very familiar with Apache principals and philosophy  for
> community
> > driven software development.
> >
> > Homogeneous Developers
> > The core developers are from Yahoo! as well as from several other
> > corporations,
> > including Cloudera and IBM.
> >
> > Reliance on Salaried Developers
> > Currently,  the developers are paid to do work on Oozie. Companies like
> > Yahoo!
> > and  Cloudera are invested in Oozie as the solution to the workflow
> >  management
> > and scheduling problem in Hadoop clusters, and that is not  likely to
> > change. In
> > addition, since workflow management is very  important for most hadoop
> > based
> > data processing, non-salaried developers  and researchers from various
> > institutes are expected to contribute to  the project.
> >
> > Relationships with Other Apache Products
> > Oozie is  based on Apache Hadoop to manage jobs created by different
> Apache
> > projects such as Hadoop, Pig, and Hive. Users of these products are
> >  extensively
> > using Oozie as their workflow scheduler.
> >
> > An Excessive Fascination with the Apache Brand
> > We deeply  respect the reputation of Apache and have had great success
> with
> > other  Apache projects such as Pig and HCatalog. We are motivated to
> expand
> > and
> > increase the adoption and development of Oozie following Apache’s
> >  established
> > open source model. We have also given reasons in the  Rationale and
> > Alignment
> > sections.
> >
> > Documentation
> > Information about Oozie can be found at http://yahoo.github.com/oozie/.
> > The
> > following links provide more information about Oozie in open source:
> >
> >        * Codebase at GitHub: https://github.com/yahoo/oozie.
> >        * JIRA : http://oozie-jira.hadoop.developer.yahoo.net
> >        * Continuous Integration (CI)  build:
> > http://oozie-ci.hadoop.developer.yahoo.net/
> >
> >        * Yahoo user community:
> > http://tech.groups.yahoo.com/group/Oozie-users/
> > Initial Source
> > Oozie has been under development since 2009 by a team of engineers at
> > Yahoo!. It
> > is currently hosted on GitHub under an Apache license at
> > https://github.com/yahoo/oozie.
> >
> > External Dependencies
> > The required  external dependencies are all Apache License or compatible
> > licenses.  Following the components with non-Apache licenses are
> > enumerated:
> >
> >        * HSQLDB License: HSQLDB
> >        * JDOM license: JDOM
> >        * BSD: Serp
> >        * CCDL v1: jaxb-api, ejb, JAF
> > NOTE:  With the exception of HSQLDB and JDOM that are directly used by
> > Oozie,
> > the other listed components are transitive dependencies of other Apache
> > components used by Oozie.
> >
> > Cryptography
> > Oozie supports the Kerberos authentication mechanism to access secured
> > Hadoop
> > services.
> >
> > Required Resources
> > Mailing Lists
> >        * oozie-private for private PMC discussions (with moderated
> > subscriptions)
> >        * oozie-dev
> >        * oozie-commits
> >        * oozie-user
> > Subversion Directory
> > https://svn.apache.org/repos/asf/incubator/oozie
> > Issue Tracking
> > JIRA Oozie (OOZIE)
> > Other Resources
> > The  existing code already has unit tests, so we would like a Hudson
> > instance
> > to run them whenever a new patch is submitted. This can be added after
> >  project
> > creation.
> >
> > Initial Committers
> >        * Mohammad K Islam (mislam77 at yahoo  dot com)
> >        * Angelo K Huang (angelohuang at gmail dot com)
> >        * Mayank Bansal (mabansal at gmail dot com)
> >        * Andreas Neumann (neunand at gmail dot com)
> >        * Alejandro Abdelnur (tucu00 at gmail dot com)
> >        * Chao Wang (brookwc at gmail dot com)
> > Affiliations
> >        * Mohammad K Islam (Yahoo!)
> >        * Angelo Huang (Yahoo!)
> >        * Mayank Bansal (Yahoo!)
> >        * Andreas Neumann (Yahoo!)
> >        * Alejandro Abdelnur (Cloudera)
> >        * Chao Wang (IBM)
> > Sponsors
> > Champion
> > Alan Gates
> > Nominated Mentors
> >        * Owen O'Malley (Incubator PMC member)
> >        * Alan Gates (Incubator PMC member)
> >        * Christopher Douglas(Incubator PMC member)
> >        * Devaraj Das (Hadoop PMC member)
> > Sponsoring EntityWe are requesting the Incubator to sponsor this project.
> >
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message