giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Avery Ching" <>
Subject Re: Review Request: Saving vertices has no status report, making it hard to find DFS issues
Date Wed, 30 Jan 2013 19:01:18 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Jan. 30, 2013, 7:01 p.m.)

Review request for giraph.


Got rid of the -1 by capturing the true reason to bypass the check (if a checkpoint is loaded).


Adds status updates to saving of the vertices and improve the overall logic of when to print
status of loading/storing the graph (every 250k vertices or 15 secs). This will help us to
see which are the slow workers when saving output. This updates the Hadoop status messages
as well as prints to the task log. I also made this consistent for the loading.

Task log messages look like the following:

INFO    2013-01-29 12:51:46,044 [main] org.apache.giraph.worker.BspServiceWorker  - saveVertices:
Saved 98751 out of 1000000 vertices, on partition 2 out of 24
INFO    2013-01-29 12:52:25,539 [main] org.apache.giraph.worker.BspServiceWorker  - saveVertices:
Saved 348752 out of 1000000 vertices, on partition 8 out of 24
INFO    2013-01-29 12:53:28,062 [main] org.apache.giraph.worker.BspServiceWorker  - saveVertices:
Saved 598753 out of 1000000 vertices, on partition 14 out of 24

I added an option for dumping output to PageRankBenchmark to test this as well.

This addresses bug GIRAPH-492.

Diffs (updated)

  giraph-core/src/main/java/org/apache/giraph/benchmark/ 3ef471a711183dd147990c5f6bb07485a58f5a71

  giraph-core/src/main/java/org/apache/giraph/bsp/ 83fba57af3f11f6c412ec59e2d121e57ea280d98

  giraph-core/src/main/java/org/apache/giraph/bsp/ 399dc72896673fbf75d2b7b933c4ea02a08f25ea

  giraph-core/src/main/java/org/apache/giraph/bsp/ 294c2c71017b5273240c9dce1f3de2be65aed289

  giraph-core/src/main/java/org/apache/giraph/graph/ d888d1038026c83b5f82956b6872eb64a44dd700

  giraph-core/src/main/java/org/apache/giraph/graph/ 401e07bb346e8ac43600992718f1f231db85aa7c

  giraph-core/src/main/java/org/apache/giraph/master/ 7ad290244946b89c6a563a86536d8e898e1c0aec

  giraph-core/src/main/java/org/apache/giraph/utils/ 81dfd1d8d9b27442543db1259235fb1825f72f7e

  giraph-core/src/main/java/org/apache/giraph/worker/ d5ad62b39517f06884ef1c003c5e91348cbe2459

  giraph-core/src/main/java/org/apache/giraph/worker/ 7522027b7306aff17668e17f4895a39e23e8a590



Passsed unittests and tested on a real cluster with PageRankBenchmark.


Avery Ching

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message