hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [DISCUSS] Fault tolerant BSP job
Date Wed, 09 Apr 2014 11:29:26 GMT
I just found this: https://issues.apache.org/jira/browse/HAMA-503 and HAMA-639.

Do you still think superstep API is essential for checkpoint/recovery?
If not, we can drop it. I don't think it's good idea.

On Wed, Apr 9, 2014 at 7:43 PM, Chia-Hung Lin <clin4j@googlemail.com> wrote:
> Not very sure if we sync at the same page. And sorry I am not very
> familiar with Superstep implementation.
>
> I assume that traditional bsp model means the original bsp interface
> where there is a bsp function and user can freely call peer.sync(),
> etc. methods
>
> .... bsp(BSPPeer ... peer) {
>     // whatever computation
>     peer.sync();
> }
>
> And the superstep style is with Superstep abstract class.
>
> If this is the case, SuperstepBSP.java has already call sync, as
> below, outside each Superstep.compute(). So it looks like even
> SuperstepPiEstimator doesn't call sync() method, barrier sync will be
> executed because each Superstep is viewed as a superstep in original
> BSP definition.
>
>   @Override
>   public void bsp(BSPPeer<K1, V1, K2, V2, M> peer) throws IOException,
>       SyncException, InterruptedException {
>     for (int index = startSuperstep; index < supersteps.length; index++) {
>       Superstep<K1, V1, K2, V2, M> superstep = supersteps[index];
>       superstep.compute(peer);
>       if (superstep.haltComputation(peer)) {
>         break;
>       }
>       peer.sync();
>       startSuperstep = 0;
>     }
>   }
>
> Within the Superstep.compute(), if sync is called again, I would think
> that another barrier sync will be executed.
>
> SuperstepBSP.java
>
> for(...) {
>   superstep .compute() -> { // in compute method
>     ...
>     peer.sync()
>   }
>   ...
>   peer.sync()
> }
>
> IIRC each call to sync may raise the checkpoint (no recovery) method
> serialize message to hdfs.
>
> For SerializePrinting, following code snippet  may move
>
> for (String otherPeer : bspPeer.getAllPeerNames()) {
>         bspPeer.send(otherPeer, new IntegerMessage(bspPeer.getPeerName(), i));
> }
>
> to Superstep.compute()
>
> And the outer for loop is what is programmed in SuperstepBSP.java
>
> for (int i = 0; i < NUM_SUPERSTEPS; i++) {
>     // code that should be moved to Superstep.compute()
> }
> bspPeer.sync();
>
>
>
> On 9 April 2014 16:17, Edward J. Yoon <edwardyoon@apache.org> wrote:
>> As you can see here[1], the sync() method never called, and an classes
>> of all superstars were needed to be declared within Job configuration.
>> Therefore, I thought it's similar with Pregel style on BSP model. It's
>> quite different from legacy model in my eyes.
>>
>> According to HAMA-505, superstep API seems used for FT job processing
>> (I didn't read closely yet). Right? In here, I have an questions. What
>> happens if I call the sync() method within compute() method? In this
>> case, framework guarantees the checkpoint/recovery? And how can I
>> implement the http://wiki.apache.org/hama/SerializePrinting using
>> superstep API?
>>
>>> What's difference between pure BSP and FT BSP? Any concrete example?
>>
>> I was mean the traditional BSP programming model.
>>
>> 1. http://svn.apache.org/repos/asf/hama/trunk/examples/src/main/java/org/apache/hama/examples/SuperstepPiEstimator.java
>>
>> On Wed, Apr 9, 2014 at 4:25 PM, Chia-Hung Lin <clin4j@googlemail.com> wrote:
>>> Sorry don't catch the point.
>>>
>>> What's difference between pure BSP and FT BSP? Any concrete example?
>>>
>>>
>>> On 9 April 2014 08:29, Edward J. Yoon <edwardyoon@apache.org> wrote:
>>>> In my eyes, SuperstepPiEstimator[1] look like totally new programming
>>>> model, very similar with Pregel.
>>>>
>>>> I personally would like to suggest that we provide both pure BSP and
>>>> fault tolerant BSP model, instead of replace.
>>>>
>>>> 1. http://svn.apache.org/repos/asf/hama/trunk/examples/src/main/java/org/apache/hama/examples/SuperstepPiEstimator.java
>>>>
>>>> --
>>>> Edward J. Yoon (@eddieyoon)
>>>> Chief Executive Officer
>>>> DataSayer, Inc.
>>
>>
>>
>> --
>> Edward J. Yoon (@eddieyoon)
>> CEO at DataSayer Co., Ltd.



-- 
Edward J. Yoon (@eddieyoon)
Chief Executive Officer
DataSayer Co., Ltd.

Mime
View raw message