giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bingjing Zhang" <jesse...@gmail.com>
Subject Re: Review Request: aggregator at input super step
Date Mon, 17 Jun 2013 16:48:27 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11525/
-----------------------------------------------------------

(Updated June 17, 2013, 4:48 p.m.)


Review request for giraph.


Changes
-------

fix some naming and typing issues


Description
-------

add aggregator for vertex reader and edge reader at input super step.

Changes in code:
add "aggregate" interface to VertexReader and EdgeReader
initialize "aggregator" in "readInputSplit" in "VertexInputSplitsCallable" and "EdgeInputSplitsCallable"
initialize master aggregator before data loading in "coordinateSuperStep" in "BspServiceMaster"
seperate master aggregator initialization and its computation for Super Step 0 (originally
they are mixed in one method "runMasterCompute")
perpare aggregator in "setup" in "BspServiceWorker" before data loading

Test:
Add SimpleVertexReader and  SimpleEdgeReader to AggregatorsTestComputation for genreating
input data.
Also add "aggregate" call in "getCurrentVertex" and "getCurrentEdge".
Register persistent aggregator with name "INPUT_VERTEX_PERSISTENT_AGG" and "INPUT_EDGE_PERSISTENT_AGG"in
AggregatorsTestMasterCompute.initialize
Assert if aggregated values can be retrived in AggregatorsTestMasterCompute.compute when "superstep
>= 0"


This addresses bug GIRAPH-673.
    https://issues.apache.org/jira/browse/GIRAPH-673


Diffs (updated)
-----

  giraph-core/src/main/java/org/apache/giraph/io/EdgeReader.java 363a5e6 
  giraph-core/src/main/java/org/apache/giraph/io/VertexReader.java b8b82af 
  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedEdgeReader.java aae7a72 
  giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedVertexReader.java 54adfec

  giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java bd48116 
  giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java eb6d30d 
  giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 78cdd8e

  giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java 977e100

  giraph-examples/src/main/java/org/apache/giraph/examples/AggregatorsTestComputation.java
db527f2 
  giraph-examples/src/main/java/org/apache/giraph/examples/GeneratedEdgeReader.java PRE-CREATION

  giraph-examples/src/test/java/org/apache/giraph/aggregators/TestAggregatorsHandling.java
6d22800 

Diff: https://reviews.apache.org/r/11525/diff/


Testing
-------

Unit Test of persistent aggregator at input super step in modified "AggregatorsTestComputation"
is done.
"mvn clean verify" was done on giraph-core
"mvn clean verfiy" was done on giraph-example

rechecked "indent" problem. Hopefully they are all solved


Thanks,

Bingjing Zhang


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message