giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bingjing Zhang" <jesse...@gmail.com>
Subject Re: Review Request: aggregator at input super step
Date Thu, 30 May 2013 16:46:30 GMT


> On May 29, 2013, 10:41 p.m., Avery Ching wrote:
> > giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java,
line 102
> > <https://reviews.apache.org/r/11525/diff/1/?file=298114#file298114line102>
> >
> >     Same as VertexInputSplitsCallable

bspServiceWorker is also used in "PartitionOwner partitionOwner = bspServiceWorker.getVertexPartitionOwner(readerVertex.getId());"
in L172-L173


> On May 29, 2013, 10:41 p.m., Avery Ching wrote:
> > giraph-core/src/main/java/org/apache/giraph/io/VertexReader.java, lines 60-70
> > <https://reviews.apache.org/r/11525/diff/1/?file=298109#file298109line60>
> >
> >     Let's do the initialize() approach.  It's cleaner.

lots of changes to initialize in subClasses


> On May 29, 2013, 10:41 p.m., Avery Ching wrote:
> > giraph-core/src/main/java/org/apache/giraph/io/EdgeReader.java, lines 59-68
> > <https://reviews.apache.org/r/11525/diff/1/?file=298108#file298108line59>
> >
> >     Let's do the initialize() approach.  It's cleaner.

lots of changes to initialize in subClasses


- Bingjing


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11525/#review21161
-----------------------------------------------------------


On May 30, 2013, 4:46 p.m., Bingjing Zhang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11525/
> -----------------------------------------------------------
> 
> (Updated May 30, 2013, 4:46 p.m.)
> 
> 
> Review request for giraph.
> 
> 
> Description
> -------
> 
> add aggregator for vertex reader and edge reader at input super step.
> 
> Changes in code:
> add "aggregate" interface to VertexReader and EdgeReader
> initialize "aggregator" in "readInputSplit" in "VertexInputSplitsCallable" and "EdgeInputSplitsCallable"
> initialize master aggregator before data loading in "coordinateSuperStep" in "BspServiceMaster"
> seperate master aggregator initialization and its computation for Super Step 0 (originally
they are mixed in one method "runMasterCompute")
> perpare aggregator in "setup" in "BspServiceWorker" before data loading
> 
> Test:
> Add SimpleVertexReader and  SimpleEdgeReader to AggregatorsTestComputation for genreating
input data.
> Also add "aggregate" call in "getCurrentVertex" and "getCurrentEdge".
> Register persistent aggregator with name "INPUT_VERTEX_PERSISTENT_AGG" and "INPUT_EDGE_PERSISTENT_AGG"in
AggregatorsTestMasterCompute.initialize
> Assert if aggregated values can be retrived in AggregatorsTestMasterCompute.compute when
"superstep >= 0"
> 
> 
> This addresses bug GIRAPH-673.
>     https://issues.apache.org/jira/browse/GIRAPH-673
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/io/EdgeReader.java 363a5e6 
>   giraph-core/src/main/java/org/apache/giraph/io/VertexReader.java 9695169 
>   giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedEdgeReader.java aae7a72

>   giraph-core/src/main/java/org/apache/giraph/io/internal/WrappedVertexReader.java 54adfec

>   giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java bd48116 
>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 8b5e39a 
>   giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java 78cdd8e

>   giraph-core/src/main/java/org/apache/giraph/worker/VertexInputSplitsCallable.java 977e100

>   giraph-examples/src/main/java/org/apache/giraph/examples/AggregatorsTestComputation.java
db527f2 
>   giraph-examples/src/test/java/org/apache/giraph/aggregators/TestAggregatorsHandling.java
6d22800 
> 
> Diff: https://reviews.apache.org/r/11525/diff/
> 
> 
> Testing
> -------
> 
> Unit Test of persistent aggregator at input super step in modified "AggregatorsTestComputation"
is done.
> "mvn clean verify" was done on giraph-core
> "mvn clean verfiy" was done on giraph-example
> 
> rechecked "indent" problem. Hopefully they are all solved
> 
> 
> Thanks,
> 
> Bingjing Zhang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message