Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A9F5C1058E for ; Mon, 12 Jan 2015 23:17:02 +0000 (UTC) Received: (qmail 26623 invoked by uid 500); 12 Jan 2015 23:17:04 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 26573 invoked by uid 500); 12 Jan 2015 23:17:04 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 26563 invoked by uid 99); 12 Jan 2015 23:17:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jan 2015 23:17:04 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of apache.mailbox@gmail.com designates 209.85.215.50 as permitted sender) Received: from [209.85.215.50] (HELO mail-la0-f50.google.com) (209.85.215.50) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Jan 2015 23:16:39 +0000 Received: by mail-la0-f50.google.com with SMTP id pn19so27093076lab.9 for ; Mon, 12 Jan 2015 15:16:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=JS4DXDylvNKs3Uo8jz5F7MkXlc0ZnltdFIOcotpHCBA=; b=Cj1WjPv58k8Csl5HkKoX31u/z1GWToNpUAucFYThqZ0eJd1yX3naJdeVcLO8MFqFjM hSTQFq5jwT+xzxObDYVXcQhGZn1Oo5L7zexiOjoDkjK0ZaJlTuQvnVpXcqONjUS3t2rt DDHcU3lNxiojw0K9wkyOvwVpJAoTM2jHmdQotYqIkkQZJK4wVD5BPILFcKoy2yKbl/an C6TF+d7RAtmgck1qptH6PKxg9e0dUw3YMCdCkk75foRG6tS4qsxAYQpdCMCQNXsXDr/t kIzEpwkdef8seCc8frgfzRSd6zFJ7vBbdvtMmRJ+ciEzLmpSjtGq6Yx403Iub9J7VjBU y/fw== MIME-Version: 1.0 X-Received: by 10.112.119.201 with SMTP id kw9mr38821680lbb.99.1421104597831; Mon, 12 Jan 2015 15:16:37 -0800 (PST) Received: by 10.152.145.132 with HTTP; Mon, 12 Jan 2015 15:16:37 -0800 (PST) Received: by 10.152.145.132 with HTTP; Mon, 12 Jan 2015 15:16:37 -0800 (PST) In-Reply-To: References: <9B36A1AE-8139-4CFD-BB4B-EBB0962005A7@yahoo-inc.com> Date: Mon, 12 Jan 2015 15:16:37 -0800 Message-ID: Subject: Re: YARN vs. MR1: is YARN a good idea? From: Eli Reisman To: user@giraph.apache.org Content-Type: multipart/alternative; boundary=047d7bae49dabe1322050c7cb219 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bae49dabe1322050c7cb219 Content-Type: text/plain; charset=UTF-8 Excellent! Hope to help out with this a bit more as time permits. I bet if we add the missing munge symbols to the hadoop_yarn profile that other error people have mentioned will go away? I know Mohammed added support for the sasl stuff in his 2.2.0 patch and I assume it still works? I'm in the middle of a large Hadoop 2.5 upgrade so maybe I can play with it for reals soon! Thanks, Eli On Dec 19, 2014 1:53 PM, "Roman Shaposhnik" wrote: > Perfect summary! Thanks for writing it. > > Thanks, > Roman. > > On Fri, Dec 19, 2014 at 12:09 PM, Eli Reisman > wrote: > > Giraph on YARN thus far doesn't break any compatibility with the > MapReduce > > version. When I was working on it more actively, it had a slightly faster > > job startup but otherwise behaved similarly to the MapReduce version. > > > > There are a number of things design wise that could make the YARN profile > > substantially better (in theory) but would require a fork or bigger > design > > changes/agreements about the MapReduce profiles. This would include > things > > like spawning the Master Giraph task in the Application Master itself, > and > > many other things along those lines. > > > > There are also a number of smaller things that would probably make a > > difference like exposing YARN's per-task resource configuration features > in > > a more flexible way. > > > > I haven't had much time to hack on Giraph this past year, and at some > point > > last summer some folks like Muhammad Islam from LinkedIn did some great > work > > to update the YARN profile to run on Hadoop 2.2.0 or newer versions but > > since then it hasn't gotten much love. > > > > I noticed there is still a note in the master POM from the original > Giraph > > on YARN implementation that says its compatible only with Hadoop > > 2.0.3-alpha. I thought that was removed with Mohammad's Hadoop 2.2.0 > patches > > but apparently it wasn't. We should remove that, it's no longer accurate > and > > seems to be misleading people trying to build the YARN profile. > > > > > > > > On Fri, Oct 10, 2014 at 11:15 AM, Tripti Singh > wrote: > >> > >> Hi Matthew, > >> I would have been thrilled to give you numbers on this one but for me > the > >> Application is not scaling without the out-of-core option( which isn't > >> working the way it was in previous version) > >> I'm still figuring it out and can get back once it's resolved. I have > >> patched a few things and will share them for people who might face > similar > >> issue. If u have a fix for scalability, do let me know > >> > >> Thanks, > >> Tripti > >> > >> Sent from my iPhone > >> > >> > On 06-Oct-2014, at 9:22 pm, "Matthew Cornell" < > matt@matthewcornell.org> > >> > wrote: > >> > > >> > Hi Folks. I don't think I paid enough attention to YARN vs. MR1 when I > >> > built Giraph 1.0.0 for our system. How much better is Giraph on YARN? > >> > Thank you. > >> > > >> > -- > >> > Matthew Cornell | matt@matthewcornell.org > --047d7bae49dabe1322050c7cb219 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Excellent! Hope to help out with this a bit more as time per= mits. I bet if we add the missing munge symbols to the hadoop_yarn profile = that other error people have mentioned will go away? I know Mohammed added = support for the sasl stuff in his 2.2.0 patch and I assume it still works?<= br> I'm in the middle of a large Hadoop 2.5 upgrade so maybe I can play wit= h it for reals soon!

Thanks,
Eli

On Dec 19, 2014 1:53 PM, "Roman Shaposhnik&= quot; <roman@shaposhnik.org&= gt; wrote:
Perfect = summary! Thanks for writing it.

Thanks,
Roman.

On Fri, Dec 19, 2014 at 12:09 PM, Eli Reisman <apache.mailbox@gmail.com> wrote:
> Giraph on YARN thus far doesn't break any compatibility with the M= apReduce
> version. When I was working on it more actively, it had a slightly fas= ter
> job startup but otherwise behaved similarly to the MapReduce version.<= br> >
> There are a number of things design wise that could make the YARN prof= ile
> substantially better (in theory) but would require a fork or bigger de= sign
> changes/agreements about the MapReduce profiles. This would include th= ings
> like spawning the Master Giraph task in the Application Master itself,= and
> many other things along those lines.
>
> There are also a number of smaller things that would probably make a > difference like exposing YARN's per-task resource configuration fe= atures in
> a more flexible way.
>
> I haven't had much time to hack on Giraph this past year, and at s= ome point
> last summer some folks like Muhammad Islam from LinkedIn did some grea= t work
> to update the YARN profile to run on Hadoop 2.2.0 or newer versions bu= t
> since then it hasn't gotten much love.
>
> I noticed there is still a note in the master POM from the original Gi= raph
> on YARN implementation that says its compatible only with Hadoop
> 2.0.3-alpha. I thought that was removed with Mohammad's Hadoop 2.2= .0 patches
> but apparently it wasn't. We should remove that, it's no longe= r accurate and
> seems to be misleading people trying to build the YARN profile.
>
>
>
> On Fri, Oct 10, 2014 at 11:15 AM, Tripti Singh <tripti@yahoo-inc.com> wrote:
>>
>> Hi Matthew,
>> I would have been thrilled to give you numbers on this one but for= me the
>> Application is not scaling without the out-of-core option( which i= sn't
>> working the way it was in previous version)
>> I'm still figuring it out and can get back once it's resol= ved. I have
>> patched a few things and will share them for people who might face= similar
>> issue. If u have a fix for scalability, do let me know
>>
>> Thanks,
>> Tripti
>>
>> Sent from my iPhone
>>
>> > On 06-Oct-2014, at 9:22 pm, "Matthew Cornell" <<= a href=3D"mailto:matt@matthewcornell.org">matt@matthewcornell.org> >> > wrote:
>> >
>> > Hi Folks. I don't think I paid enough attention to YARN v= s. MR1 when I
>> > built Giraph 1.0.0 for our system. How much better is Giraph = on YARN?
>> > Thank you.
>> >
>> > --
>> > Matthew Cornell | = matt@matthewcornell.org
--047d7bae49dabe1322050c7cb219--