giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (GIRAPH-13) Port Giraph to YARN
Date Fri, 01 Mar 2013 01:22:13 GMT

     [ https://issues.apache.org/jira/browse/GIRAPH-13?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eli Reisman updated GIRAPH-13:
------------------------------

    Attachment: GIRAPH-13-7.patch

All the YARN plumbing works now, I am (for working purposes) hardcoding our examples jar as
my artifact to bring into YARN/HDFS for the job run, but already have a working mechanism
to include jars of your choosing that I will polish up when the "review-worthy" patch is up.

Also made lots of progress getting my dummy Mapper#Context into shape to fool Giraph. That
and some tests are the last (forseeable) blockers here. Will post another patch (hopefully
ready to review) very soon.

Any ideas/comments are appreciated. I ended up having to hand-exclude the o.a.g.yarn package
from all profiles but hadoop_yarn since the classes I use are only available by depending
on "hadoop-yarn-common" and I didn't want to import that for all the profiles.

My attempt to simply exclude by default the yarn package and then re-include it only in my
profile, while pretty on the screen, does not make Maven happy for some reason. Will play
with it some more, but I think its a "feature" of the path filtering in Maven and I have not
seen documented or found a proper fix yet. Will keep looking, but this works fine as-is for
now.

OK, comments/ideas/criticisms welcome. Thanks.



                
> Port Giraph to YARN
> -------------------
>
>                 Key: GIRAPH-13
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-13
>             Project: Giraph
>          Issue Type: New Feature
>            Reporter: Jakob Homan
>            Assignee: Eli Reisman
>         Attachments: GIRAPH-13-1.patch, GIRAPH-13-2.patch, GIRAPH-13-3.patch, GIRAPH-13-4.patch,
GIRAPH-13-5.patch, GIRAPH-13-6.patch, GIRAPH-13-7.patch
>
>
> Now that YARN (aka MR2 aka MAPREDUCE-279) has been merged into the Hadoop trunk, we should
think about what it would take to separate out the graph processing bits of Giraph from the
MR1-specific code so as to take advantage of the less-MR centric aspects of YARN, while still
supporting both over the medium term.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message