giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Edunov" <>
Subject Re: Review Request 23989: Improve checkpointing
Date Tue, 12 Aug 2014 00:54:00 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Aug. 12, 2014, 12:53 a.m.)

Review request for giraph.


Addressing CR issues

Repository: giraph-git


We need to address some issues with checkpointing:
1) worker2worker messages are not saved
2) BspServiceWorker does not compile under hadoop_0.23 profile
3) it would be nice to be able to manually checkpoint and stop any job at any point of time.


1) worker2worker messages fixed my serializing currentworkertoworker messages (it is a list
of writable so I had to write class information as well)
2) Compilation issues fixed
3) The way you can trigger checkpointing now is by creating /_checkpointAndStop node in zookeeper
(same way as _haltComputation works) After that the behavior of the job will be determined
by registered GiraphJobRetryChecker. By default, job will get checkpointed at the end of current
superstep and halted. You can override this behavior by making shouldRestartCheckpoint() return
true, in this case job will be restarted immediately after getting checkpointed.

Diffs (updated)

  giraph-core/src/main/java/org/apache/giraph/bsp/ 02577b9 
  giraph-core/src/main/java/org/apache/giraph/bsp/ ff3e427 
  giraph-core/src/main/java/org/apache/giraph/bsp/ e5b7cf3 
  giraph-core/src/main/java/org/apache/giraph/bsp/ e5d0ae1 
  giraph-core/src/main/java/org/apache/giraph/bsp/ PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/bsp/ c384fbf 
  giraph-core/src/main/java/org/apache/giraph/comm/ a92cd1c 
  giraph-core/src/main/java/org/apache/giraph/conf/ 0424a47 
  giraph-core/src/main/java/org/apache/giraph/graph/ c351778 
  giraph-core/src/main/java/org/apache/giraph/graph/ bc56c9c 
  giraph-core/src/main/java/org/apache/giraph/graph/ 6ebb002 
  giraph-core/src/main/java/org/apache/giraph/job/ 0cab86c

  giraph-core/src/main/java/org/apache/giraph/job/ 4a1f02e 
  giraph-core/src/main/java/org/apache/giraph/job/ 53a800e 
  giraph-core/src/main/java/org/apache/giraph/job/ 9530fd6 
  giraph-core/src/main/java/org/apache/giraph/master/ e129390 
  giraph-core/src/main/java/org/apache/giraph/master/ 0635210 
  giraph-core/src/main/java/org/apache/giraph/utils/ PRE-CREATION 
  giraph-core/src/main/java/org/apache/giraph/utils/ 763f59d 
  giraph-core/src/main/java/org/apache/giraph/worker/ d2d24ee 
  giraph-core/src/test/java/org/apache/giraph/utils/ PRE-CREATION 
  giraph-examples/src/test/java/org/apache/giraph/ 2939af7 
  pom.xml ed2a98c 



Run pagerank, will keep testing with different jobs.


Sergey Edunov

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message