giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Phillip Rhodes <motley.crue....@gmail.com>
Subject [SOLVED] Re: Giraph job never ends
Date Thu, 12 Mar 2015 03:54:10 GMT
OK, this was easy enough to fix, once I understood what
was actually happening.  Since I'm running on EC2 nodes on
AWS, it is not the case that any give node can talk to any other
node on any port (at least not by default).  I had tried to
cherry-pick which ports to whitelist in the security group,
but I missed one or more that YARN needed for internal
communication.   I discovered this when examining the
resourcemanager logs.


For now, instead of trying to enumerate exactly which ports
to allow, I added a rule to allow "all traffic" for address 10.0.0.0/24
and that solved this.


Cheers,


Phil


On Wed, Mar 11, 2015 at 1:39 PM, Phillip Rhodes
<motley.crue.fan@gmail.com> wrote:
> Interesting... It totally did not work for me when built using the
> hadoop_2 profile, but with the hadoop_yarn profile everything at least
> starts up.  I'm pretty baffled right now... my cluster is essentially
> working, and I can run, for example, the WordCount example just fine.
> And the Giraph job starts and shows no apparent errors, but I get no
> output and it seems to run forever.
>
> It's probably some really small detail of my Hadoop configuration, or
> some environmental issue.  The problem is, I don't even know where to
> start looking right now.  :-(
>
>
> Phil
> This message optimized for indexing by NSA PRISM
>
>
> On Wed, Mar 11, 2015 at 3:16 AM, Martin Junghanns
> <martin.junghanns@gmx.net> wrote:
>> Hi Phillip,
>>
>> I am using Hadoop 2.5.2 with Giraph 1.1.0 and it runs fine with
>> -Phadoop2 (from scratch) and -Phadoop_yarn (after removing
>> STATIC_SASL_SYMBOL from munge.symbols in pom.xml).
>>
>> Maybe you can also try the stable Giraph
>> version and report your problem as an issue?
>>
>> Cheers,
>> Martin
>>
>> On 11.03.2015 04:03, Phillip Rhodes wrote:
>>> Giraph crew:
>>>
>>> I'm trying to run the SimpleShortestPathsComputation example using
>>> the latest Giraph code and Hadoop 2.5.2.  My command line looks
>>> like this:
>>>
>>> hadoop jar
>>> /home/prhodes/giraph/giraph-examples/target/giraph-examples-1.2.0-SNAPSHOT-for-hadoop-2.5.2-jar-with-dependencies.jar
>>>
>>>
>> org.apache.giraph.GiraphRunner
>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>>>
>>>
>> -vip /user/prhodes/input/tiny_graph.txt -vof
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>>> /user/prhodes/giraph_output/shortestpaths -w 4
>>>
>>>
>>> and the job appears to start OK.  But then it starts outputing
>>> these kinds of messages, and this just continues (seemingly)
>>> forever until you ctrl+c it.
>>>
>>> 15/03/11 02:54:31 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation,
>>> Elapsed: 305.43 secs 15/03/11 02:54:31 INFO yarn.GiraphYarnClient:
>>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers
>>> used: 1 15/03/11 02:54:35 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation,
>>> Elapsed: 309.44 secs 15/03/11 02:54:35 INFO yarn.GiraphYarnClient:
>>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers
>>> used: 1 15/03/11 02:54:39 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation,
>>> Elapsed: 313.45 secs 15/03/11 02:54:39 INFO yarn.GiraphYarnClient:
>>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers
>>> used: 1 15/03/11 02:54:43 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation,
>>> Elapsed: 317.45 secs 15/03/11 02:54:43 INFO yarn.GiraphYarnClient:
>>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers
>>> used: 1 ^C15/03/11 02:54:47 INFO yarn.GiraphYarnClient: Giraph:
>>> org.apache.giraph.examples.SimpleShortestPathsComputation,
>>> Elapsed: 321.46 secs 15/03/11 02:54:47 INFO yarn.GiraphYarnClient:
>>> appattempt_1426041786848_0002_000001, State: ACCEPTED, Containers
>>> used: 1
>>>
>>> Any idea what is going on here?
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Phil ---
>>>
>>>
>>> This message optimized for indexing by NSA PRISM
>>>

Mime
View raw message