giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (GIRAPH-13) Port Giraph to YARN
Date Sat, 02 Mar 2013 02:15:17 GMT

     [ https://issues.apache.org/jira/browse/GIRAPH-13?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eli Reisman updated GIRAPH-13:
------------------------------

    Attachment: GIRAPH-13-8.patch

This version runs to completion (as in output gets written) when running examples on pure
yarn profile. If you have 2.0.3-alpha installed, build giraph like:
{code}
mvn -Phadoop_yarn clean package
{code}

then use the giraph-examples jar with deps to run using a command line such as this:

{code}
bin/hadoop --config etc/hadoop jar share/hadoop/giraph/giraph-examples-0.2-SNAPSHOT-for-hadoop-2.0.3-alpha-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex -w 1 -vif
org.apache.giraph.io.formats.IntIntNullIntTextInputFormat -of org.apache.giraph.io.formats.IdWithValueTextOutputFormat
-vip /user/ereisman/graph -op /user/ereisman/output
{code}

A lot of the functionality we need (CLI opts etc) is already there, you can run your own jars
or include whatever you want in the job, and it will run on cluster too.

It still needs a lot of cleanup, tests, and I have to get the YARN setup code to end the job
nicely (fail or success) and I need the output committed to the right place. And I will remove
the hardcoded dep on giraph-examples ;) All that will be in the next patch. 

But, this is working and will commit Giraph output to HDFS. All setStatus msgs end up in the
logs for now. You may have to CTRL-C out of the Yarn Client when its done (for now but not
for long...)

More to follow...

                
> Port Giraph to YARN
> -------------------
>
>                 Key: GIRAPH-13
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-13
>             Project: Giraph
>          Issue Type: New Feature
>            Reporter: Jakob Homan
>            Assignee: Eli Reisman
>         Attachments: GIRAPH-13-1.patch, GIRAPH-13-2.patch, GIRAPH-13-3.patch, GIRAPH-13-4.patch,
GIRAPH-13-5.patch, GIRAPH-13-6.patch, GIRAPH-13-7.patch, GIRAPH-13-8.patch
>
>
> Now that YARN (aka MR2 aka MAPREDUCE-279) has been merged into the Hadoop trunk, we should
think about what it would take to separate out the graph processing bits of Giraph from the
MR1-specific code so as to take advantage of the less-MR centric aspects of YARN, while still
supporting both over the medium term.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message