Mailing-List: contact dev-help@giraph.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@giraph.apache.org
Content-Type: multipart/alternative;
 boundary="===============8144906083855039882=="
MIME-Version: 1.0
Subject: Re: Review Request: GIRAPH-13: Port Giraph to YARN
From: "Eli Reisman" <initialcontext@gmail.com>
To: "Eli Reisman" <initialcontext@gmail.com>, "giraph" <dev@giraph.apache.org>
Date: Sat, 09 Mar 2013 02:37:00 -0000
Message-ID: <20130309023700.27129.16828@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Eli Reisman" <noreply@reviews.apache.org>
References: <20130308033220.27098.22288@reviews.apache.org>
In-Reply-To: <20130308033220.27098.22288@reviews.apache.org>
Reply-To: "Eli Reisman" <initialcontext@gmail.com>

--===============8144906083855039882==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9811/
-----------------------------------------------------------

(Updated March 9, 2013, 2:36 a.m.)


Review request for giraph.


Changes
-------

Just an update. Latency issues taken care of, just need to decide on a stab=
le version to support and finish the integration test. Almost done!


Description
-------

Port Giraph to "pure YARN" clusters, using Hadoop MapReduce classes in our =
code (IO formats etc.) but running the cluster job without any active parti=
cipation by a running MapReduce framework. This means doing some things our=
selves that Hadoop used to do for us.

I am putting this up for review to aid some non-Giraphers in having a peek =
at the YARN component. There is a bit of latency in the job launch that I a=
m still diagnosing. I am also still finishing up an integration test to ver=
ify the YARN components can run a no-op Giraph job successfully. All BSP co=
de is covered by our MRv1 tests, which are sufficient since once Giraph is =
running, it does not know or care if its running on YARN. The grand total i=
s TWO files with FOUR actual munges, total for the entire patch. All the re=
st is conditionally compiled and/or manipulated through conf settings witho=
ut ever calling into YARN-specific code from inside Giraph. This will allow=
 us to wait on ripping apart our IO formats or other MRv1 baked-in dependen=
cies before we're ready to abandon MR. This also sets up a paradigm by whic=
h it will be easy to port us to other cluster frameworks (Mesos, etc.)

I will ping Giraph folks when this is really ready for review (hopefully ne=
xt day or so) but feel free to drop me a line now if you see something you =
are curious about or just plain don't like. The sooner I fix it, the sooner=
 this gets committed, so please speak up if you do.

My goal is to make this not only our port of YARN, but another (there aren'=
t many) good and well-commented example of how to run "real applications" l=
ike Giraph on YARN clusters. So I'm hoping its clear and easy to follow on =
that level as well. Happy to hear feedback on that angle as well!

Thanks! Will post a wiki page explaining a bit more about this when its all=
 finished. This version is still depending on Hadoop-2.0.3-alpha, but I wil=
l attempt to back port to 2.0.2 before I'm done, and a future JIRA should b=
ring us to 2.0.0 or higher (and trunk of course.)
 =


Diffs (updated)
-----

  checkstyle.xml 10a8b43 =

  giraph-core/pom.xml 2a500d8 =

  giraph-core/src/main/java/org/apache/giraph/GiraphRunner.java 5bd5686 =

  giraph-core/src/main/java/org/apache/giraph/bsp/BspInputFormat.java bce84=
b1 =

  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java=
 6886d58 =

  giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java ad9=
073d =

  giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java e=
74c59a =

  giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java =
b84e21b =

  giraph-core/src/main/java/org/apache/giraph/utils/ConfigurationUtils.java=
 41238d0 =

  giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java =
74c1f87 =

  giraph-core/src/main/java/org/apache/giraph/yarn/GiraphApplicationMaster.=
java PRE-CREATION =

  giraph-core/src/main/java/org/apache/giraph/yarn/GiraphYarnClient.java PR=
E-CREATION =

  giraph-core/src/main/java/org/apache/giraph/yarn/GiraphYarnTask.java PRE-=
CREATION =

  giraph-core/src/main/java/org/apache/giraph/yarn/YarnUtils.java PRE-CREAT=
ION =

  giraph-core/src/main/java/org/apache/giraph/yarn/package-info.java PRE-CR=
EATION =

  giraph-core/src/test/java/org/apache/giraph/yarn/TestYarnJob.java PRE-CRE=
ATION =

  giraph-examples/pom.xml 7a18711 =

  pom.xml a65208d =


Diff: https://reviews.apache.org/r/9811/diff/


Testing
-------

Getting there, in-progress integration test is included for your amusment.


Thanks,

Eli Reisman


--===============8144906083855039882==--