giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roman Shaposhnik <ro...@shaposhnik.org>
Subject Re: YARN vs. MR1: is YARN a good idea?
Date Fri, 19 Dec 2014 21:53:28 GMT
Perfect summary! Thanks for writing it.

Thanks,
Roman.

On Fri, Dec 19, 2014 at 12:09 PM, Eli Reisman <apache.mailbox@gmail.com> wrote:
> Giraph on YARN thus far doesn't break any compatibility with the MapReduce
> version. When I was working on it more actively, it had a slightly faster
> job startup but otherwise behaved similarly to the MapReduce version.
>
> There are a number of things design wise that could make the YARN profile
> substantially better (in theory) but would require a fork or bigger design
> changes/agreements about the MapReduce profiles. This would include things
> like spawning the Master Giraph task in the Application Master itself, and
> many other things along those lines.
>
> There are also a number of smaller things that would probably make a
> difference like exposing YARN's per-task resource configuration features in
> a more flexible way.
>
> I haven't had much time to hack on Giraph this past year, and at some point
> last summer some folks like Muhammad Islam from LinkedIn did some great work
> to update the YARN profile to run on Hadoop 2.2.0 or newer versions but
> since then it hasn't gotten much love.
>
> I noticed there is still a note in the master POM from the original Giraph
> on YARN implementation that says its compatible only with Hadoop
> 2.0.3-alpha. I thought that was removed with Mohammad's Hadoop 2.2.0 patches
> but apparently it wasn't. We should remove that, it's no longer accurate and
> seems to be misleading people trying to build the YARN profile.
>
>
>
> On Fri, Oct 10, 2014 at 11:15 AM, Tripti Singh <tripti@yahoo-inc.com> wrote:
>>
>> Hi Matthew,
>> I would have been thrilled to give you numbers on this one but for me the
>> Application is not scaling without the out-of-core option( which isn't
>> working the way it was in previous version)
>> I'm still figuring it out and can get back once it's resolved. I have
>> patched a few things and will share them for people who might face similar
>> issue. If u have a fix for scalability, do let me know
>>
>> Thanks,
>> Tripti
>>
>> Sent from my iPhone
>>
>> > On 06-Oct-2014, at 9:22 pm, "Matthew Cornell" <matt@matthewcornell.org>
>> > wrote:
>> >
>> > Hi Folks. I don't think I paid enough attention to YARN vs. MR1 when I
>> > built Giraph 1.0.0 for our system. How much better is Giraph on YARN?
>> > Thank you.
>> >
>> > --
>> > Matthew Cornell | matt@matthewcornell.org

Mime
View raw message