Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 57A4C17403 for ; Sat, 14 Mar 2015 02:28:38 +0000 (UTC) Received: (qmail 34249 invoked by uid 500); 14 Mar 2015 02:28:38 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 34200 invoked by uid 500); 14 Mar 2015 02:28:38 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 34188 invoked by uid 99); 14 Mar 2015 02:28:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Mar 2015 02:28:37 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [74.125.149.238] (HELO na3sys009aog115.obsmtp.com) (74.125.149.238) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Mar 2015 02:28:32 +0000 Received: from mail-we0-f171.google.com ([74.125.82.171]) (using TLSv1) by na3sys009aob115.postini.com ([74.125.148.12]) with SMTP ID DSNKVQOb/38V3uGAMALQrGtaQMOFcv/C9/XG@postini.com; Fri, 13 Mar 2015 19:28:12 PDT Received: by wejb47 with SMTP id b47so3661485wej.0 for ; Fri, 13 Mar 2015 19:25:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=3nuycY3xIAApP6fXP3HFfcyNR8S8AP2qVFZGYJ9hJ4U=; b=BvHO/zpZTr0Nz7XCq2TxvLgMMoABnWY9W+cQjHZBCzitr2Duz0uGltApZVrBisFV0N yzabo9oNVT+1BAY5fBxBaZFVlWu2KW7dhiCLsmzsoZYhTX7sPFZuUxRxwBLAs1tTiYje 735KY//Iz6gQ67bG003GVi0C6EpQmGal91Gq6DuQrjbRJvgTZkniamSWtMuAB6MDAPRS b7kuCKPNg4JcOTEU1OZloJPHR2Yvv2CL1nytzVaRZdesCZb89uAhCsCiFHKOhPp2u8Fe npkXrSy63SRclXboQAAsbPM88AI6+IBO9o7wZ/KxbiPTUjTWXz+dv9P/J2DpyU8PNens /AAA== X-Gm-Message-State: ALoCoQmLDu9bAysygQKnipxQuKMVU0FESRtmnJaQ72NwOgBygwkc7LIgv/UfWpPnk3DOF7G7EtDuSGjREAudtQkDGw10XcuMQnHk5pQUEQDrbn9UpNswMCdKovfpx3D1LkljUK5t3RARsCFzYFGplzxA3QNgpUPo8w== X-Received: by 10.180.104.200 with SMTP id gg8mr69909651wib.8.1426299901860; Fri, 13 Mar 2015 19:25:01 -0700 (PDT) X-Received: by 10.180.104.200 with SMTP id gg8mr69909637wib.8.1426299901679; Fri, 13 Mar 2015 19:25:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.27.21.193 with HTTP; Fri, 13 Mar 2015 19:24:41 -0700 (PDT) In-Reply-To: References: From: Steven Harenberg Date: Fri, 13 Mar 2015 22:24:41 -0400 Message-ID: Subject: Re: [SOLVED] Re: Giraph job never ends To: user Content-Type: multipart/alternative; boundary=f46d04430404fbb48d05113652e9 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04430404fbb48d05113652e9 Content-Type: text/plain; charset=UTF-8 Thanks Phil, I appreciate the help. Your posts over the past couple days have already been quite helpful. There were a few things I was going to play with as well, perhaps it is some configuration issue as you mentioned earlier. I had some issues with EC2 today and I will look at it again tomorrow. Thanks for letting me know about your talk, it sounds interesting. I will try and go as long as I can get there in time. --Steve On Fri, Mar 13, 2015 at 3:37 PM, Phillip Rhodes wrote: > Steve: > > I'm not 100% sure what to tell you, and I don't have access to my > cluster right this minute. But later this evening I can log in and > see if I can find anything that might be > useful to you. > > Also, as an FYI, I'll be doing a presentation on Giraph at the > Triangle Java User's Group meeting this coming Monday... if you're in > the area (I see you have an @ncsu.edu address), and you can come by, I > might be able to help you then. Part of my presentation will be > walking through how to setup a Giraph / YARN cluster, based on my > experiences over the past few days... > > > Phil > > This message optimized for indexing by NSA PRISM > > > On Fri, Mar 13, 2015 at 3:30 PM, Steven Harenberg > wrote: > > Hey Phil, > > > > I have been having the exact same problems as you (I am also setting up > > Giraph on EC2), but this solution did not work for me. > > > > Do you recall what error you saw in resourcemanager logs? I am also > looking > > at these logs, but nothing is standing out to me. In fact, it almost > seems > > like the application should have successfully finished. The log stops > > updating and I see a lot of "COMPLETED", "RESULT=SUCCESS", "FINISHED" at > the > > end of the log. Though, it does look like one of the containers is not > > transitioning to these states. > > > > Thanks, > > Steve > > > > > > On Wed, Mar 11, 2015 at 11:54 PM, Phillip Rhodes < > motley.crue.fan@gmail.com> > > wrote: > >> > >> OK, this was easy enough to fix, once I understood what > >> was actually happening. Since I'm running on EC2 nodes on > >> AWS, it is not the case that any give node can talk to any other > >> node on any port (at least not by default). I had tried to > >> cherry-pick which ports to whitelist in the security group, > >> but I missed one or more that YARN needed for internal > >> communication. I discovered this when examining the > >> resourcemanager logs. > >> > >> > >> For now, instead of trying to enumerate exactly which ports > >> to allow, I added a rule to allow "all traffic" for address 10.0.0.0/24 > >> and that solved this. > >> > >> > >> Cheers, > >> > >> > >> Phil > >> > >> > >> On Wed, Mar 11, 2015 at 1:39 PM, Phillip Rhodes > >> wrote: > >> > Interesting... It totally did not work for me when built using the > >> > hadoop_2 profile, but with the hadoop_yarn profile everything at least > >> > starts up. I'm pretty baffled right now... my cluster is essentially > >> > working, and I can run, for example, the WordCount example just fine. > >> > And the Giraph job starts and shows no apparent errors, but I get no > >> > output and it seems to run forever. > >> > > >> > It's probably some really small detail of my Hadoop configuration, or > >> > some environmental issue. The problem is, I don't even know where to > >> > start looking right now. :-( > >> > > >> > > >> > Phil > >> > This message optimized for indexing by NSA PRISM > >> > > >> > > >> > On Wed, Mar 11, 2015 at 3:16 AM, Martin Junghanns > >> > wrote: > >> >> Hi Phillip, > >> >> > >> >> I am using Hadoop 2.5.2 with Giraph 1.1.0 and it runs fine with > >> >> -Phadoop2 (from scratch) and -Phadoop_yarn (after removing > >> >> STATIC_SASL_SYMBOL from munge.symbols in pom.xml). > >> >> > >> >> Maybe you can also try the stable Giraph > >> >> version and report your problem as an issue? > >> >> > >> >> Cheers, > >> >> Martin > >> >> > >> >> On 11.03.2015 04:03, Phillip Rhodes wrote: > >> >>> Giraph crew: > >> >>> > >> >>> I'm trying to run the SimpleShortestPathsComputation example using > >> >>> the latest Giraph code and Hadoop 2.5.2. My command line looks > >> >>> like this: > >> >>> > >> >>> hadoop jar > >> >>> > >> >>> > /home/prhodes/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar > >> >>> > >> >>> > >> >> org.apache.giraph.GiraphRunner > >> >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif > >> >>> > >> >>> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat > >> >>> > >> >>> > >> >> -vip /user/prhodes/input/tiny_graph.txt -vof > >> >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > >> >>> /user/prhodes/giraph_output/shortestpaths -w 4 > >> >>> > >> >>> > >> >>> and the job appears to start OK. But then it starts outputing > >> >>> these kinds of messages, and this just continues (seemingly) > >> >>> forever until you ctrl+c it. > >> >>> > >> >>> 15/03/11 02:54:31 INFO yarn.GiraphYarnClient: Giraph: > >> >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >> >>> Elapsed: 305.43 secs 15/03/11 02:54:31 INFO yarn.GiraphYarnClient: > >> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >> >>> used: 1 15/03/11 02:54:35 INFO yarn.GiraphYarnClient: Giraph: > >> >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >> >>> Elapsed: 309.44 secs 15/03/11 02:54:35 INFO yarn.GiraphYarnClient: > >> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >> >>> used: 1 15/03/11 02:54:39 INFO yarn.GiraphYarnClient: Giraph: > >> >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >> >>> Elapsed: 313.45 secs 15/03/11 02:54:39 INFO yarn.GiraphYarnClient: > >> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >> >>> used: 1 15/03/11 02:54:43 INFO yarn.GiraphYarnClient: Giraph: > >> >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >> >>> Elapsed: 317.45 secs 15/03/11 02:54:43 INFO yarn.GiraphYarnClient: > >> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >> >>> used: 1 ^C15/03/11 02:54:47 INFO yarn.GiraphYarnClient: Giraph: > >> >>> org.apache.giraph.examples.SimpleShortestPathsComputation, > >> >>> Elapsed: 321.46 secs 15/03/11 02:54:47 INFO yarn.GiraphYarnClient: > >> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers > >> >>> used: 1 > >> >>> > >> >>> Any idea what is going on here? > >> >>> > >> >>> > >> >>> Thanks, > >> >>> > >> >>> > >> >>> Phil --- > >> >>> > >> >>> > >> >>> This message optimized for indexing by NSA PRISM > >> >>> > > > > > --f46d04430404fbb48d05113652e9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Phil, I appreciate the help. Your posts over the pa= st couple days have already been quite helpful.

There we= re a few things I was going to play with as well, perhaps it is some config= uration issue as you mentioned earlier. I had some issues with EC2 today an= d I will look at it again tomorrow.

Thanks for letting me kno= w about your talk, it sounds interesting. I will try and go as long as I ca= n get there in time.

--Steve

On Fri, Mar 13, 2015 at 3:3= 7 PM, Phillip Rhodes <motley.crue.fan@gmail.com> wro= te:
Steve:

I'm not 100% sure what to tell you, and I don't have access to my cluster right this minute.=C2=A0 But later this evening I can log in and see if I can find anything that might be
useful to you.

Also, as an FYI, I'll be doing a presentation on Giraph at the
Triangle Java User's Group meeting this coming Monday... if you're = in
the area (I see you have an @= ncsu.edu address), and you can come by, I
might be able to help you then.=C2=A0 =C2=A0Part of my presentation will be=
walking through how to setup a Giraph / YARN cluster, based on my
experiences over the past few days...


Phil

This message optimized for indexing by NSA PRISM


On Fri, Mar 13, 2015 at 3:30 PM, Steven Harenberg <sdharenb@ncsu.edu>= ; wrote:
> Hey Phil,
>
> I have been having the exact same problems as you (I am also setting u= p
> Giraph on EC2), but this solution did not work for me.
>
> Do you recall what error you saw in resourcemanager logs? I am also lo= oking
> at these logs, but nothing is standing out to me. In fact, it almost s= eems
> like the application should have successfully finished. The log stops<= br> > updating and I see a lot of "COMPLETED", "RESULT=3DSUCC= ESS", "FINISHED" at the
> end of the log. Though, it does look like one of the containers is not=
> transitioning to these states.
>
> Thanks,
> Steve
>
>
> On Wed, Mar 11, 2015 at 11:54 PM, Phillip Rhodes <motley.crue.fan@gmail.com= >
> wrote:
>>
>> OK, this was easy enough to fix, once I understood what
>> was actually happening.=C2=A0 Since I'm running on EC2 nodes o= n
>> AWS, it is not the case that any give node can talk to any other >> node on any port (at least not by default).=C2=A0 I had tried to >> cherry-pick which ports to whitelist in the security group,
>> but I missed one or more that YARN needed for internal
>> communication.=C2=A0 =C2=A0I discovered this when examining the >> resourcemanager logs.
>>
>>
>> For now, instead of trying to enumerate exactly which ports
>> to allow, I added a rule to allow "all traffic" for addr= ess 10.0.0.0/24
>> and that solved this.
>>
>>
>> Cheers,
>>
>>
>> Phil
>>
>>
>> On Wed, Mar 11, 2015 at 1:39 PM, Phillip Rhodes
>> <motley.crue.fan@gmail.com> wrote:
>> > Interesting... It totally did not work for me when built usin= g the
>> > hadoop_2 profile, but with the hadoop_yarn profile everything= at least
>> > starts up.=C2=A0 I'm pretty baffled right now... my clust= er is essentially
>> > working, and I can run, for example, the WordCount example ju= st fine.
>> > And the Giraph job starts and shows no apparent errors, but I= get no
>> > output and it seems to run forever.
>> >
>> > It's probably some really small detail of my Hadoop confi= guration, or
>> > some environmental issue.=C2=A0 The problem is, I don't e= ven know where to
>> > start looking right now.=C2=A0 :-(
>> >
>> >
>> > Phil
>> > This message optimized for indexing by NSA PRISM
>> >
>> >
>> > On Wed, Mar 11, 2015 at 3:16 AM, Martin Junghanns
>> > <martin.junghanns@gmx.net> wrote:
>> >> Hi Phillip,
>> >>
>> >> I am using Hadoop 2.5.2 with Giraph 1.1.0 and it runs fin= e with
>> >> -Phadoop2 (from scratch) and -Phadoop_yarn (after removin= g
>> >> STATIC_SASL_SYMBOL from munge.symbols in pom.xml).
>> >>
>> >> Maybe you can also try the stable Giraph
>> >> version and report your problem as an issue?
>> >>
>> >> Cheers,
>> >> Martin
>> >>
>> >> On 11.03.2015 04:03, Phillip Rhodes wrote:
>> >>> Giraph crew:
>> >>>
>> >>> I'm trying to run the SimpleShortestPathsComputat= ion example using
>> >>> the latest Giraph code and Hadoop 2.5.2.=C2=A0 My com= mand line looks
>> >>> like this:
>> >>>
>> >>> hadoop jar
>> >>>
>> >>> /home/prhodes/giraph/giraph-examples/target/giraph-ex= amples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar
>> >>>
>> >>>
>> >> org.apache.giraph.GiraphRunner
>> >>> org.apache.giraph.examples.SimpleShortestPathsComputa= tion -vif
>> >>>
>> >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubl= eVertexInputFormat
>> >>>
>> >>>
>> >> -vip /user/prhodes/input/tiny_graph.txt -vof
>> >>> org.apache.giraph.io.formats.IdWithValueTextOutputFor= mat -op
>> >>> /user/prhodes/giraph_output/shortestpaths -w 4
>> >>>
>> >>>
>> >>> and the job appears to start OK.=C2=A0 But then it st= arts outputing
>> >>> these kinds of messages, and this just continues (see= mingly)
>> >>> forever until you ctrl+c it.
>> >>>
>> >>> 15/03/11 02:54:31 INFO yarn.GiraphYarnClient: Giraph:=
>> >>> org.apache.giraph.examples.SimpleShortestPathsComputa= tion,
>> >>> Elapsed: 305.43 secs 15/03/11 02:54:31 INFO yarn.Gira= phYarnClient:
>> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED= , Containers
>> >>> used: 1 15/03/11 02:54:35 INFO yarn.GiraphYarnClient:= Giraph:
>> >>> org.apache.giraph.examples.SimpleShortestPathsComputa= tion,
>> >>> Elapsed: 309.44 secs 15/03/11 02:54:35 INFO yarn.Gira= phYarnClient:
>> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED= , Containers
>> >>> used: 1 15/03/11 02:54:39 INFO yarn.GiraphYarnClient:= Giraph:
>> >>> org.apache.giraph.examples.SimpleShortestPathsComputa= tion,
>> >>> Elapsed: 313.45 secs 15/03/11 02:54:39 INFO yarn.Gira= phYarnClient:
>> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED= , Containers
>> >>> used: 1 15/03/11 02:54:43 INFO yarn.GiraphYarnClient:= Giraph:
>> >>> org.apache.giraph.examples.SimpleShortestPathsComputa= tion,
>> >>> Elapsed: 317.45 secs 15/03/11 02:54:43 INFO yarn.Gira= phYarnClient:
>> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED= , Containers
>> >>> used: 1 ^C15/03/11 02:54:47 INFO yarn.GiraphYarnClien= t: Giraph:
>> >>> org.apache.giraph.examples.SimpleShortestPathsComputa= tion,
>> >>> Elapsed: 321.46 secs 15/03/11 02:54:47 INFO yarn.Gira= phYarnClient:
>> >>> appattempt_1426041786848_0002_000001, State: ACCEPTED= , Containers
>> >>> used: 1
>> >>>
>> >>> Any idea what is going on here?
>> >>>
>> >>>
>> >>> Thanks,
>> >>>
>> >>>
>> >>> Phil ---
>> >>>
>> >>>
>> >>> This message optimized for indexing by NSA PRISM
>> >>>
>
>

--f46d04430404fbb48d05113652e9--