incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyunsik Choi <hyun...@apache.org>
Subject Re: [PROPOSAL] Proposing Giraph for the Apache Incubator
Date Sun, 17 Jul 2011 15:15:04 GMT

I read the proposal of Giraph. That's very interesting! I have some
experiences in graph processing with MapReduce. I think that the approach of
Giraph is very promising since MapReduce already is regarded as the de-facto
standards in processing large data and is appropriate to many graph
algorithms. If there is a general graph package in MapReduce, it will be
widely used in many areas. I would like to participate in Giraph project.

Best regards,
Hyunsik Choi


Avery Ching wrote:
> 
> Hi,
> 
> I would like to propose Giraph as an Apache Incubator project.  Giraph is
> a large-scale graph processing infrastructure (inspired by Pregel) that
> runs entirely on Hadoop.  Giraph applications and MapReduce jobs coexist
> on shared Hadoop instances and Giraph applications can be part of Oozie
> workflows as a normal MapReduce job.
> 
> Here is a link to the proposal in our GitHub wiki:
> 
> https://github.com/aching/Giraph/wiki/Apache-Incubator-Proposal
> 
> The proposal is also inlined below:
> 
> Thanks!
> 
> Avery
> 
> 
> 
> = Giraph : Large-scale graph processing on Hadoop =
> 
> == Abstract ==
> 
> Giraph is a large-scale, fault-tolerant, Bulk Synchronous Parallel
> (BSP)-based graph processing framework.
> 
> == Proposal ==
> 
> Graph processing platforms to run large-scale algorithms (such as page
> rank, shared connections, personalization-based popularity, etc.) have
> become quite popular.  Some recent examples include Pregel and HaLoop. 
> For general-purpose big data computation, the MapReduce computation model
> is widely adopted and the most deployed MapReduce infrastructure is Apache
> Hadoop.  We have implemented a graph-processing framework that is launched
> as a typical Hadoop MapReduce job to leverage existing Hadoop
> infrastructure, such as Amazon’s EC2.  Giraph builds upon the
> graph-oriented nature of Pregel but additionally adds fault-tolerance to
> the coordinator process with the use of ZooKeeper as its centralized
> coordination service.  Additionally, Giraph will include a library of
> generic graph algorithms.
> 
> == Background ==
> 
> Giraph was initially began development as a side project at Yahoo! at the
> end of 2010.  It was made functional in a month and then started adding
> various features.  Development has been focused on internal customers
> needs until this point.
> 
> == Rationale ==
> 
> Web and online social graphs have been rapidly growing in size and scale
> during the past decade.  In 2008, Google estimated that the number of web
> pages reached over a trillion.  Online social networking and email sites,
> including Yahoo!, Google, Microsoft, Facebook, LinkedIn, and Twitter, have
> hundreds of millions of users and are expected to grow much more in the
> future.  Processing these graphs plays a big role in relevant and
> personalized information for users, such as results from a search engine
> or news in an online social networking site.
> 
> == Initial Goals ==
> 
> At this point, most of the functionality has been implemented and we are
> looking to get more adoption and contributions from users outside Yahoo!.  
> We want to ensure that performance scales and that the code is robust and
> fault tolerant.
> 
> == Current Status ==
> 
> === Meritocracy ===
> 
> Giraph was initially developed by Avery Ching and Christian Kunz beginning
> in December 2010 at Yahoo!.  There are other developers using Giraph at
> Yahoo! that are making suggestions and adding code.  We are reaching out
> to other folks at social networking companies for additional usage and
> development.
> 
> === Community ===
> 
> Several groups who are interested in either joining our project or using
> our code have contacted us.  We certainly believe that there is a lot of
> interest and are actively looking to improve and expand the community.
> 
> === Core Developers ===
> 
> Avery Ching: Wrote a majority of the code
> Christian Kunz: Wrote most of the communication code and security
> integration with Hadoop
> 
> === Alignment ===
> 
> Giraph uses several Apache projects as its underlying infrastructure
> (Hadoop and ZooKeeper).   It also builds on Apache Maven.
> 
> == Known Risks ==
> 
> === Orphaned products ===
> 
> There are many social networking companies that would be interested in
> using this graph-processing framework and we have already received
> interest from some of them.  Yahoo! is already using this code in
> production and will certainly continue to use it in the future as well.
> 
> === Inexperience with Open Source ===
> 
> While the initial developers have limited experience on contributing to
> open-source projects, Yahoo! as a company has a strong commitment to
> open-source and we have several advisors that we can ask for help.
> 
> === Homogenous Developers ===
> 
> At this time, the project is relatively young and the developers work at
> only two companies (Yahoo! and Jybe).  However, given the interest we have
> seen in the project, we expect the diversity to improve in the near
> future.
> 
> === Reliance on Salaried Developers ===
> 
> Currently Giraph is being developed by a combination of salaried and
> volunteer time.  We expect that other corporations will take an interest
> in this project and likely contribute with salaried developers.  Some
> individuals will likely spend volunteer time on it as well.  It is still
> early in their project and we are hoping for a lot of growth.
> 
> === Relationships with Other Apache Products ===
> 
> Giraph depends on many Apache projects: Hadoop, ZooKeeper, Log4j, Commons,
> etc.  It is built using Apache Maven.
> 
> Giraph has some overlapping functionality with Apache Hama.  However,
> there are some significant differences.  Giraph focuses on graph-based
> bulk synchronous parallel (BSP) computing, while Apache Hama is more for
> general purposed BSP computing.  Giraph runs on the Hadoop infrastructure,
> while Apache Hama uses its own computing framework.
> 
> === An Excessive Fascination with the Apache Brand ===
> 
> The Apache brand is likely to help us find contributors, however, our
> interests in Apache are primarily because the other projects that we
> depend on are also Apache projects and it makes sense that all this
> software be available from the same place.
> 
> === Documentation ===
> 
> Currently we have little documentation, but several examples.  We are
> working on improving this situation.
> 
> === Initial Source ===
> 
> The initial source of the code is from Yahoo! and began development in
> December 2010.  It is already available on GitHub at
> https://github.com/aching/Giraph.
> 
> === Source and Intellectual Property Submission Plan ===
> 
> We intend the entire code base to be licensed under the Apache License,
> Version 2.0.
> 
> === External Dependencies ===
> 
> The required dependencies are all Apache compatible licenses.  The
> following components with non-Apache licenses are enumerated:
> * JSON – Public Domain
> 
> === Cryptography ===
> 
> Giraph depends on secure Hadoop that can optionally use Kerberos.
> 
> == Required Resources ==
> 
> === Mailing lists ===
> 
> * giraph-private (with moderated subscriptions)
> * giraph-dev
> * giraph-commits
> * giraph-users
> 
> === Subversion Directory ===
> 
> https://svn.apache.org/repos/asf/incubator/giraph
> 
> === Issue Tracking ===
> 
> JIRA Giraph (GIRAPH)
> 
> === Other Resources ===
> 
> Giraph has integration tests that can be run with the LocalJobRunner. 
> These same tests also designed to be run on a small (even single node)
> Hadoop cluster.  While not required at this time, it would be nice if such
> a resource were available.
> 
> === Initial Committers ===
> 
> Avery Ching, aching at yahoo-inc dot com
> Christian Kunz, christian at jybe-inc dot com
> Owen O’Malley, owen at hortonworks dot com
> 
> === Affiliations ===
> 
> Avery Ching, Yahoo!
> Christian Kunz, Jybe
> 
> == Sponsors ==
> 
> === Champion ===
> 
> Owen O’ Malley
> 
> === Nominated Mentors ===
> 
> Owen O’Malley
> 
> === Sponsoring Entity ===
> 
> Apache Incubator PMC
> 
> 

-- 
View this message in context: http://old.nabble.com/-PROPOSAL--Proposing-Giraph-for-the-Apache-Incubator-tp32070326p32078057.html
Sent from the Apache Incubator - General mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message