giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maja Kabiljo" <majakabi...@fb.com>
Subject Re: Review Request 23140: Fix checkpointing
Date Mon, 30 Jun 2014 23:03:14 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23140/#review47023
-----------------------------------------------------------


I see a lot of the changes are related to aggregators, and you write them now from master,
worker and MasterCompute - can't we write them just once and go through normal path of distributing
them in the beginning of the superstep?

- Maja Kabiljo


On June 27, 2014, 8:48 p.m., Sergey Edunov wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23140/
> -----------------------------------------------------------
> 
> (Updated June 27, 2014, 8:48 p.m.)
> 
> 
> Review request for giraph.
> 
> 
> Repository: giraph-git
> 
> 
> Description
> -------
> 
> This fix merely makes checkpointing work again. 
> 
> 
> Diffs
> -----
> 
>   giraph-core/src/main/java/org/apache/giraph/aggregators/Aggregator.java 514e470 
>   giraph-core/src/main/java/org/apache/giraph/aggregators/AggregatorHandler.java PRE-CREATION

>   giraph-core/src/main/java/org/apache/giraph/aggregators/AggregatorWrapper.java 9613805

>   giraph-core/src/main/java/org/apache/giraph/aggregators/BasicAggregator.java 07a4100

>   giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java 2e35373 
>   giraph-core/src/main/java/org/apache/giraph/comm/ServerData.java f0ecca2 
>   giraph-core/src/main/java/org/apache/giraph/comm/aggregators/AllAggregatorServerData.java
177e738 
>   giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java 7d7ceb2 
>   giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java ad7e045 
>   giraph-core/src/main/java/org/apache/giraph/master/DefaultMasterCompute.java bfb6f0e

>   giraph-core/src/main/java/org/apache/giraph/master/MasterAggregatorHandler.java 325d91f

>   giraph-core/src/main/java/org/apache/giraph/master/MasterCompute.java d77a9b5 
>   giraph-core/src/main/java/org/apache/giraph/master/WritableMasterAggregatorUsage.java
PRE-CREATION 
>   giraph-core/src/main/java/org/apache/giraph/partition/BasicPartitionOwner.java 545d1af

>   giraph-core/src/main/java/org/apache/giraph/partition/HashMasterPartitioner.java 240687e

>   giraph-core/src/main/java/org/apache/giraph/partition/HashWorkerPartitioner.java d833895

>   giraph-core/src/main/java/org/apache/giraph/partition/MasterGraphPartitioner.java 50c750a

>   giraph-core/src/main/java/org/apache/giraph/partition/PartitionBalancer.java 3454d62

>   giraph-core/src/main/java/org/apache/giraph/partition/PartitionOwner.java 0ac74da 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimpleMasterPartitioner.java
f128f34 
>   giraph-core/src/main/java/org/apache/giraph/partition/SimpleWorkerPartitioner.java
3c0de44 
>   giraph-core/src/main/java/org/apache/giraph/partition/WorkerGraphPartitioner.java 004ea81

>   giraph-core/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 09dd46d

>   giraph-core/src/main/java/org/apache/giraph/utils/io/ExtendedDataInputOutput.java af45426

>   giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java 8dcf19a 
>   giraph-core/src/main/java/org/apache/giraph/worker/WorkerAggregatorHandler.java 9bfd7b5

>   giraph-core/src/main/java/org/apache/giraph/worker/WorkerContext.java 17347db 
>   giraph-core/src/main/java/org/apache/giraph/worker/WorkerThreadAggregatorUsage.java
194127e 
>   giraph-core/src/main/java/org/apache/giraph/worker/WritableWorkerAggregatorUsage.java
PRE-CREATION 
>   giraph-core/src/test/java/org/apache/giraph/partition/SimpleRangePartitionFactoryTest.java
96bd5d7 
>   giraph-examples/src/test/java/org/apache/giraph/aggregators/TestAggregatorsHandling.java
e2b611b 
> 
> Diff: https://reviews.apache.org/r/23140/diff/
> 
> 
> Testing
> -------
> 
> I tested it running multiple different jobs. I run page rank on 2*10^9 vertices on 200
workers and it seems to work just fine. It only takes 2 minutes to save checkpoint. 
> 
> 
> Thanks,
> 
> Sergey Edunov
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message