I have tried increase number of reties to 10. But it is the same: no retry at all. Isn't retrying of failed task the default behavior for hadoop? Why isn't it working in the case of Giraph?

Here is the message from master:

2013-03-18 18:23:30,628 ERROR org.apache.giraph.graph.BspServiceMaster: checkWorkers: Did not receive enough processes in time (only 54 of 55 required).  This occurs if you do not have enough map tasks available simultaneously on your Hadoop instance to fulfill the number of requested workers.
2013-03-18 18:23:30,628 FATAL org.apache.giraph.graph.BspServiceMaster: coordinateSuperstep: Not enough healthy workers for superstep 12
2013-03-18 18:23:30,629 INFO org.apache.giraph.graph.BspServiceMaster: setJobState: {"_stateKey":"FAILED","_applicationAttemptKey":-1,"_superstepKey":-1} on superstep 12
2013-03-18 18:23:30,649 FATAL org.apache.giraph.graph.BspServiceMaster: failJob: Killing job job_201303181655_0004
2013-03-18 18:23:30,703 FATAL org.apache.giraph.graph.GraphMapper: uncaughtException: OverrideExceptionHandler on thread org.apache.giraph.graph.MasterThread, msg = null, exiting...
                at org.apache.giraph.graph.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1411)
                at org.apache.giraph.graph.MasterThread.run(MasterThread.java:111)
2013-03-18 18:23:30,705 WARN org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Forced a shutdown hook kill of the ZooKeeper process.

The workers except for the one who threw the expected exception report the following error:

2013-03-18 18:20:54,107 ERROR org.apache.zookeeper.ClientCnxn: Error while calling watcher
java.lang.RuntimeException: process: Disconnected from ZooKeeper, cannot recover - WatchedEvent state:Disconnected type:None path:null
                at org.apache.giraph.graph.BspService.process(BspService.java:974)
                at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
                at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
2013-03-18 18:20:55,110 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server idp30.almaden.ibm.com/
2013-03-18 18:20:55,111 WARN org.apache.zookeeper.ClientCnxn: Session 0x13d8037f8100008 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
                at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119)
2013-03-18 18:20:55,218 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-03-18 18:20:55,254 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2013-03-18 18:20:55,254 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName ytian for UID 3005 from the native implementation
2013-03-18 18:20:55,257 WARN org.apache.hadoop.mapred.Child: Error running child
java.lang.IllegalStateException: startSuperstep: KeeperException getting assignments
                at org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:928)
                at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:649)
                at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:891)
                at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
                at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:396)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
                at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /_hadoopBsp/job_201303181655_0004/_applicationAttemptsDir/0/_superstepDir/2/_partitionAssignments
                at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
                at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
                at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
                at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
                at org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:909)


From:        Avery Ching <aching@apache.org>
To:        user@giraph.apache.org
Cc:        Yuanyuan Tian/Almaden/IBM@IBMUS
Date:        03/18/2013 03:05 PM
Subject:        Re: about fault tolerance in Giraph

How many retries did you set for hadoop map task failures?  Might want to try 10?


On 3/18/13 2:38 PM, Yuanyuan Tian wrote:

Hi Avery,

I was just testing how Giraph can handle fault tolerance. I wrote a simple algorithm that could run without a problem. Then I artificially added a line of code to throw an IOException for the 12th superstep when the taskID is the 0001 and attempt ID is 0000. The job returned the excepted IOException, but it cannot recover from it. There is no retry of the failed task, even though there are empty map slots left in the cluster. Eventually, the whole job failed after time out.


Avery Ching <aching@apache.org>
03/18/2013 02:09 PM
Re: about fault tolerance in Giraph

Hi Yuanyuan,

We haven't tested this feature in a while.  But it should work.  What did the job report about why it failed?


On 3/18/13 10:22 AM, Yuanyuan Tian wrote:

Can anyone help me answer the question?


Yuanyuan Tian/Almaden/IBM@IBMUS
03/15/2013 02:05 PM
about fault tolerance in Giraph


I was testing the fault tolerance of Giraph on a long running job. I noticed that when one of the worker throw an exception, the whole job failed without retrying the task, even though I turned on the checkpointing and there were available map slots in my cluster. Why wasn't the fault tolerance mechanism working?

I was running a version of Giraph downloaded sometime in June 2012 and I used Netty for the communication layer.