giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jyotirmoy Sundi <sundi...@gmail.com>
Subject Re: error while running giraph of YARN (was working in MR1)
Date Mon, 03 Mar 2014 06:14:20 GMT
Hmmm,
I am running in cloudera manager. The number of application master,master
and worker seems as per the config stats.

I get the following response in master mapper while running on a small graph
MASTER_ONLY checkWorkers: Only found 2 responses of 3 needed to start
superstep -1

When I go to the mapper running master, I get the following log:

INFO org.apache.giraph.master.BspServiceMaster:
logMissingWorkersOnSuperstep: No response from partition 3 (could be
master)

Any idea what configuration issue it might be ?


Thanks

Sundi




On Sun, Mar 2, 2014 at 4:56 PM, Eli Reisman <apache.mailbox@gmail.com>wrote:

> This looks like YARN cluster is misconfigured. Alternately, you need to
> configure it to allow a few more worker tasks. Giraph on YARN at minimum
> needs one Application Master, one Master, and one Worker (so 3 YARN
> containers) I have a feeling this could be the issue.
>
>
> On Sat, Mar 1, 2014 at 9:18 PM, Jyotirmoy Sundi <sundi133@gmail.com>wrote:
>
>> Hi Folks,
>>
>>     The job was working properly in MR1 without any issue. I am trying to run a simple
CC sample Giraph job on YARN. . I have attached the stacktrace and a few errors. Any pointers
will be really helpful for the below errors.
>>
>> *1. BspServiceMaster (YARN profile) is FAILING this task, throwing exception to end
job run.*
>>
>> *2. java.lang.IllegalStateException: Not enough healthy workers to create input splits*
>>
>>
>>
>>
>> *StackTrace:*
>>
>> 2014-03-02 04:53:24,646 INFO org.apache.giraph.master.BspServiceMaster: logMissingWorkersOnSuperstep:
No response from partition 2 (could be master)
>> 2014-03-02 04:53:24,646 ERROR org.apache.giraph.master.BspServiceMaster: checkWorkers:
Did not receive enough processes in time (only 1 of 2 required) after waiting 600000msecs).
 This occurs if you do not have enough map tasks available simultaneously on your Hadoop instance
to fulfill the number of requested workers.
>> 2014-03-02 04:53:24,649 INFO org.apache.giraph.master.BspServiceMaster: setJobState:
{"_stateKey":"FAILED","_applicationAttemptKey":-1,"_superstepKey":-1} on superstep -1
>> 2014-03-02 04:53:24,653 FATAL org.apache.giraph.master.BspServiceMaster: failJob:
Killing job job_201402281650_0019
>> 2014-03-02 04:53:24,654 FATAL org.apache.giraph.master.BspServiceMaster: failJob:
exception java.lang.IllegalStateException: Not enough healthy workers to create input splits
>> 2014-03-02 04:53:24,654 ERROR org.apache.giraph.master.MasterThread: masterThread:
Master algorithm failed with RuntimeException
>> java.lang.RuntimeException: BspServiceMaster (YARN profile) is FAILING this task,
throwing exception to end job run.
>> 	at org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:349)
>> 	at org.apache.giraph.master.BspServiceMaster.setJobStateFailed(BspServiceMaster.java:297)
>> 	at org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:616)
>> 	at org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:692)
>> 	at org.apache.giraph.master.MasterThread.run(MasterThread.java:100)
>> Caused by: java.lang.IllegalStateException: Not enough healthy workers to create
input splits
>> 	... 4 more
>> 2014-03-02 04:53:24,656 FATAL org.apache.giraph.graph.GraphMapper: uncaughtException:
OverrideExceptionHandler on thread org.apache.giraph.master.MasterThread, msg = java.lang.RuntimeException:
BspServiceMaster (YARN profile) is FAILING this task, throwing exception to end job run.,
exiting...
>> java.lang.IllegalStateException: java.lang.RuntimeException: BspServiceMaster (YARN
profile) is FAILING this task, throwing exception to end job run.
>> 	at org.apache.giraph.master.MasterThread.run(MasterThread.java:181)
>> Caused by: java.lang.RuntimeException: BspServiceMaster (YARN profile) is FAILING
this task, throwing exception to end job run.
>> 	at org.apache.giraph.master.BspServiceMaster.failJob(BspServiceMaster.java:349)
>> 	at org.apache.giraph.master.BspServiceMaster.setJobStateFailed(BspServiceMaster.java:297)
>> 	at org.apache.giraph.master.BspServiceMaster.createInputSplits(BspServiceMaster.java:616)
>> 	at org.apache.giraph.master.BspServiceMaster.createVertexInputSplits(BspServiceMaster.java:692)
>> 	at org.apache.giraph.master.MasterThread.run(MasterThread.java:100)
>> Caused by: java.lang.IllegalStateException: Not enough healthy workers to create
input splits
>> 	... 4 more
>>
>> ------------------------------
>>
>>
>


-- 
Best Regards,
Jyotirmoy Sundi

Mime
View raw message