giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Reisman <apache.mail...@gmail.com>
Subject Re: YARN vs. MR1: is YARN a good idea?
Date Fri, 19 Dec 2014 20:09:50 GMT
Giraph on YARN thus far doesn't break any compatibility with the MapReduce
version. When I was working on it more actively, it had a slightly faster
job startup but otherwise behaved similarly to the MapReduce version.

There are a number of things design wise that could make the YARN profile
substantially better (in theory) but would require a fork or bigger design
changes/agreements about the MapReduce profiles. This would include things
like spawning the Master Giraph task in the Application Master itself, and
many other things along those lines.

There are also a number of smaller things that would probably make a
difference like exposing YARN's per-task resource configuration features in
a more flexible way.

I haven't had much time to hack on Giraph this past year, and at some point
last summer some folks like Muhammad Islam from LinkedIn did some great
work to update the YARN profile to run on Hadoop 2.2.0 or newer versions
but since then it hasn't gotten much love.

I noticed there is still a note in the master POM from the original Giraph
on YARN implementation that says its compatible only with Hadoop
2.0.3-alpha. I thought that was removed with Mohammad's Hadoop 2.2.0
patches but apparently it wasn't. We should remove that, it's no longer
accurate and seems to be misleading people trying to build the YARN profile.



On Fri, Oct 10, 2014 at 11:15 AM, Tripti Singh <tripti@yahoo-inc.com> wrote:
>
> Hi Matthew,
> I would have been thrilled to give you numbers on this one but for me the
> Application is not scaling without the out-of-core option( which isn't
> working the way it was in previous version)
> I'm still figuring it out and can get back once it's resolved. I have
> patched a few things and will share them for people who might face similar
> issue. If u have a fix for scalability, do let me know
>
> Thanks,
> Tripti
>
> Sent from my iPhone
>
> > On 06-Oct-2014, at 9:22 pm, "Matthew Cornell" <matt@matthewcornell.org>
> wrote:
> >
> > Hi Folks. I don't think I paid enough attention to YARN vs. MR1 when I
> > built Giraph 1.0.0 for our system. How much better is Giraph on YARN?
> > Thank you.
> >
> > --
> > Matthew Cornell | matt@matthewcornell.org
>

Mime
View raw message