giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman" <initialcont...@gmail.com>
Subject Re: Review Request: Saving vertices has no status report, making it hard to find DFS issues
Date Wed, 30 Jan 2013 17:51:27 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9138/#review15871
-----------------------------------------------------------


Maybe put in a named constant where you're using a -1 flag a few places, but this is a big
+1 from me, its just a nit, up to you. This will be very helpful!

- Eli Reisman


On Jan. 29, 2013, 10:34 p.m., Avery Ching wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9138/
> -----------------------------------------------------------
> 
> (Updated Jan. 29, 2013, 10:34 p.m.)
> 
> 
> Review request for giraph.
> 
> 
> Description
> -------
> 
> Adds status updates to saving of the vertices and improve the overall logic of when to
print status of loading/storing the graph (every 250k vertices or 15 secs). This will help
us to see which are the slow workers when saving output. This updates the Hadoop status messages
as well as prints to the task log. I also made this consistent for the loading.
> 
> Task log messages look like the following:
> 
> INFO    2013-01-29 12:51:46,044 [main] org.apache.giraph.worker.BspServiceWorker  - saveVertices:
Saved 98751 out of 1000000 vertices, on partition 2 out of 24
> INFO    2013-01-29 12:52:25,539 [main] org.apache.giraph.worker.BspServiceWorker  - saveVertices:
Saved 348752 out of 1000000 vertices, on partition 8 out of 24
> INFO    2013-01-29 12:53:28,062 [main] org.apache.giraph.worker.BspServiceWorker  - saveVertices:
Saved 598753 out of 1000000 vertices, on partition 14 out of 24
> 
> I added an option for dumping output to PageRankBenchmark to test this as well.
> 
> 
> This addresses bug GIRAPH-492.
>     https://issues.apache.org/jira/browse/GIRAPH-492
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java 3ef471a711183dd147990c5f6bb07485a58f5a71

>   giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedService.java 83fba57af3f11f6c412ec59e2d121e57ea280d98

>   giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java 399dc72896673fbf75d2b7b933c4ea02a08f25ea

>   giraph-core/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 294c2c71017b5273240c9dce1f3de2be65aed289

>   giraph-core/src/main/java/org/apache/giraph/graph/FinishedSuperstepStats.java d888d1038026c83b5f82956b6872eb64a44dd700

>   giraph-core/src/main/java/org/apache/giraph/graph/GraphTaskManager.java 401e07bb346e8ac43600992718f1f231db85aa7c

>   giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java 7ad290244946b89c6a563a86536d8e898e1c0aec

>   giraph-core/src/main/java/org/apache/giraph/utils/LoggerUtils.java 81dfd1d8d9b27442543db1259235fb1825f72f7e

>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java d5ad62b39517f06884ef1c003c5e91348cbe2459

>   giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java 7522027b7306aff17668e17f4895a39e23e8a590

> 
> Diff: https://reviews.apache.org/r/9138/diff/
> 
> 
> Testing
> -------
> 
> Passsed unittests and tested on a real cluster with PageRankBenchmark.
> 
> 
> Thanks,
> 
> Avery Ching
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message