giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <>
Subject [jira] [Updated] (GIRAPH-13) Port Giraph to YARN
Date Fri, 01 Mar 2013 01:22:13 GMT


Eli Reisman updated GIRAPH-13:

    Attachment: GIRAPH-13-7.patch

All the YARN plumbing works now, I am (for working purposes) hardcoding our examples jar as
my artifact to bring into YARN/HDFS for the job run, but already have a working mechanism
to include jars of your choosing that I will polish up when the "review-worthy" patch is up.

Also made lots of progress getting my dummy Mapper#Context into shape to fool Giraph. That
and some tests are the last (forseeable) blockers here. Will post another patch (hopefully
ready to review) very soon.

Any ideas/comments are appreciated. I ended up having to hand-exclude the o.a.g.yarn package
from all profiles but hadoop_yarn since the classes I use are only available by depending
on "hadoop-yarn-common" and I didn't want to import that for all the profiles.

My attempt to simply exclude by default the yarn package and then re-include it only in my
profile, while pretty on the screen, does not make Maven happy for some reason. Will play
with it some more, but I think its a "feature" of the path filtering in Maven and I have not
seen documented or found a proper fix yet. Will keep looking, but this works fine as-is for

OK, comments/ideas/criticisms welcome. Thanks.

> Port Giraph to YARN
> -------------------
>                 Key: GIRAPH-13
>                 URL:
>             Project: Giraph
>          Issue Type: New Feature
>            Reporter: Jakob Homan
>            Assignee: Eli Reisman
>         Attachments: GIRAPH-13-1.patch, GIRAPH-13-2.patch, GIRAPH-13-3.patch, GIRAPH-13-4.patch,
GIRAPH-13-5.patch, GIRAPH-13-6.patch, GIRAPH-13-7.patch
> Now that YARN (aka MR2 aka MAPREDUCE-279) has been merged into the Hadoop trunk, we should
think about what it would take to separate out the graph processing bits of Giraph from the
MR1-specific code so as to take advantage of the less-MR centric aspects of YARN, while still
supporting both over the medium term.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message