giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: [jira] [Commented] (GIRAPH-13) Port Giraph to YARN
Date Wed, 03 Apr 2013 13:55:23 GMT
Thanks Avery! I learned a ton doing it. As far as I could tell there were
some claims on the web about YARN implementations in the works that were
dated when I got to them. The only real examples I found were from
hadoop-common, and were bare-bones-minimum (dist. shell) or overly complex
(MRv2!) as a YARN reference. The other is Spark/Shark which is wonderful,
but Scala end-to-end so not an ideal example app for all new YARNers. So it
is likely we are the first Java project outside of Hadoop itself that is
not a trivial application of YARN.

It was hard for me during the writing of the YARN-specific parts not to get
crazy ideas. There are so many options it gives us that were not possible
on MRv1. I am excited to see what sort of ideas the community will cook up
to take advantage of YARN as we go forward.

Anyway, I'm getting set up on a new machine and will run some more data
through it and begin to strategize about a few more things I'd like to do
in the short term as I have time. But the cool part is, at this point, I
think its easy enough to follow that folks can jump in from here and extend
it with no trouble. Anyone that wants too, feel free to bug me with
questions along the way. There are also a few odd gotchas that I will
document on the wiki soon.



On Wed, Apr 3, 2013 at 4:25 AM, Hudson (JIRA) <jira@apache.org> wrote:

>
>     [
> https://issues.apache.org/jira/browse/GIRAPH-13?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13620721#comment-13620721]
>
> Hudson commented on GIRAPH-13:
> ------------------------------
>
> Integrated in Giraph-trunk-Commit #862 (See [
> https://builds.apache.org/job/Giraph-trunk-Commit/862/])
>     GIRAPH-13: Port Giraph to YARN (Revision
> b2dff2751d8d3d768f788b39089688c18f6c1750)
>
>      Result = SUCCESS
> ereisman :
> http://git-wip-us.apache.org/repos/asf?p=giraph.git&a=commit&h=b2dff2751d8d3d768f788b39089688c18f6c1750
> Files :
> * giraph-core/src/main/java/org/apache/giraph/yarn/GiraphYarnClient.java
> * giraph-core/src/main/java/org/apache/giraph/yarn/package-info.java
> * pom.xml
> * giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java
> * giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java
> * giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java
> * giraph-core/src/test/resources/capacity-scheduler.xml
> * giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java
> * giraph-core/pom.xml
> * giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java
> * giraph-core/src/test/java/org/apache/giraph/yarn/TestYarnJob.java
> *
> giraph-core/src/main/java/org/apache/giraph/yarn/GiraphApplicationMaster.java
> * giraph-core/src/main/java/org/apache/giraph/yarn/GiraphYarnTask.java
> * giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java
> * giraph-core/src/main/java/org/apache/giraph/yarn/YarnUtils.java
> * CHANGELOG
> * giraph-examples/pom.xml
> * checkstyle.xml
> * giraph-core/src/main/java/org/apache/giraph/GiraphRunner.java
> * giraph-core/src/main/java/org/apache/giraph/bsp/BspInputFormat.java
>
>
> > Port Giraph to YARN
> > -------------------
> >
> >                 Key: GIRAPH-13
> >                 URL: https://issues.apache.org/jira/browse/GIRAPH-13
> >             Project: Giraph
> >          Issue Type: New Feature
> >            Reporter: Jakob Homan
> >            Assignee: Eli Reisman
> >         Attachments: GIRAPH-13-1.patch, GIRAPH-13-2.patch,
> GIRAPH-13-3.patch, GIRAPH-13-4.patch, GIRAPH-13-5.patch, GIRAPH-13-6.patch,
> GIRAPH-13-7.patch, GIRAPH-13-8.patch, GIRAPH-13-9.patch,
> GIRAPH-13-9-r1.patch, GIRAPH-13-9-r2.patch, GIRAPH-13-9-r3.patch,
> GIRAPH-13-9-r4.patch, GIRAPH-13-9-r5.patch, GIRAPH-13-9-r6.patch
> >
> >
> > Now that YARN (aka MR2 aka MAPREDUCE-279) has been merged into the
> Hadoop trunk, we should think about what it would take to separate out the
> graph processing bits of Giraph from the MR1-specific code so as to take
> advantage of the less-MR centric aspects of YARN, while still supporting
> both over the medium term.
> > Review Board link (ready for review now):
> https://reviews.apache.org/r/9811/
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA
> administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message