giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bingjing Zhang" <jesse...@gmail.com>
Subject Re: Review Request: aggregator at input super step
Date Wed, 12 Jun 2013 03:25:50 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11525/
-----------------------------------------------------------

(Updated June 12, 2013, 3:25 a.m.)


Review request for giraph.


Changes
-------

update comments, remove unnecessary functions.


Description
-------

add aggregator for vertex reader and edge reader at input super step.

Changes in code:
add "aggregate" interface to VertexReader and EdgeReader
initialize "aggregator" in "readInputSplit" in "VertexInputSplitsCallable" and "EdgeInputSplitsCallable"
initialize master aggregator before data loading in "coordinateSuperStep" in "BspServiceMaster"
seperate master aggregator initialization and its computation for Super Step 0 (originally
they are mixed in one method "runMasterCompute")
perpare aggregator in "setup" in "BspServiceWorker" before data loading

Test:
Add SimpleVertexReader and  SimpleEdgeReader to AggregatorsTestComputation for genreating
input data.
Also add "aggregate" call in "getCurrentVertex" and "getCurrentEdge".
Register persistent aggregator with name "INPUT_VERTEX_PERSISTENT_AGG" and "INPUT_EDGE_PERSISTENT_AGG"in
AggregatorsTestMasterCompute.initialize
Assert if aggregated values can be retrived in AggregatorsTestMasterCompute.compute when "superstep
>= 0"


This addresses bug GIRAPH-673.
    https://issues.apache.org/jira/browse/GIRAPH-673


Diffs (updated)
-----

  giraph-core/src/main/java/org/apache/giraph/io/EdgeReader.java 363a5e6 
  giraph-core/src/main/java/org/apache/giraph/io/VertexReader.java 9695169 
  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedEdgeReader.java aae7a72 
  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedVertexReader.java 54adfec

  giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java bd48116 
  giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 8b5e39a 
  giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 78cdd8e

  giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java 977e100

  giraph-examples/src/main/java/org/apache/giraph/examples/AggregatorsTestComputation.java
db527f2 
  giraph-examples/src/test/java/org/apache/giraph/aggregators/TestAggregatorsHandling.java
6d22800 

Diff: https://reviews.apache.org/r/11525/diff/


Testing
-------

Unit Test of persistent aggregator at input super step in modified "AggregatorsTestComputation"
is done.
"mvn clean verify" was done on giraph-core
"mvn clean verfiy" was done on giraph-example

rechecked "indent" problem. Hopefully they are all solved


Thanks,

Bingjing Zhang


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message