hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chia-Hung Lin <cli...@googlemail.com>
Subject Re: [DISCUSS] Fault tolerant BSP job
Date Wed, 09 Apr 2014 10:43:40 GMT
Not very sure if we sync at the same page. And sorry I am not very
familiar with Superstep implementation.

I assume that traditional bsp model means the original bsp interface
where there is a bsp function and user can freely call peer.sync(),
etc. methods

.... bsp(BSPPeer ... peer) {
    // whatever computation
    peer.sync();
}

And the superstep style is with Superstep abstract class.

If this is the case, SuperstepBSP.java has already call sync, as
below, outside each Superstep.compute(). So it looks like even
SuperstepPiEstimator doesn't call sync() method, barrier sync will be
executed because each Superstep is viewed as a superstep in original
BSP definition.

  @Override
  public void bsp(BSPPeer<K1, V1, K2, V2, M> peer) throws IOException,
      SyncException, InterruptedException {
    for (int index = startSuperstep; index < supersteps.length; index++) {
      Superstep<K1, V1, K2, V2, M> superstep = supersteps[index];
      superstep.compute(peer);
      if (superstep.haltComputation(peer)) {
        break;
      }
      peer.sync();
      startSuperstep = 0;
    }
  }

Within the Superstep.compute(), if sync is called again, I would think
that another barrier sync will be executed.

SuperstepBSP.java

for(...) {
  superstep .compute() -> { // in compute method
    ...
    peer.sync()
  }
  ...
  peer.sync()
}

IIRC each call to sync may raise the checkpoint (no recovery) method
serialize message to hdfs.

For SerializePrinting, following code snippet  may move

for (String otherPeer : bspPeer.getAllPeerNames()) {
        bspPeer.send(otherPeer, new IntegerMessage(bspPeer.getPeerName(), i));
}

to Superstep.compute()

And the outer for loop is what is programmed in SuperstepBSP.java

for (int i = 0; i < NUM_SUPERSTEPS; i++) {
    // code that should be moved to Superstep.compute()
}
bspPeer.sync();



On 9 April 2014 16:17, Edward J. Yoon <edwardyoon@apache.org> wrote:
> As you can see here[1], the sync() method never called, and an classes
> of all superstars were needed to be declared within Job configuration.
> Therefore, I thought it's similar with Pregel style on BSP model. It's
> quite different from legacy model in my eyes.
>
> According to HAMA-505, superstep API seems used for FT job processing
> (I didn't read closely yet). Right? In here, I have an questions. What
> happens if I call the sync() method within compute() method? In this
> case, framework guarantees the checkpoint/recovery? And how can I
> implement the http://wiki.apache.org/hama/SerializePrinting using
> superstep API?
>
>> What's difference between pure BSP and FT BSP? Any concrete example?
>
> I was mean the traditional BSP programming model.
>
> 1. http://svn.apache.org/repos/asf/hama/trunk/examples/src/main/java/org/apache/hama/examples/SuperstepPiEstimator.java
>
> On Wed, Apr 9, 2014 at 4:25 PM, Chia-Hung Lin <clin4j@googlemail.com> wrote:
>> Sorry don't catch the point.
>>
>> What's difference between pure BSP and FT BSP? Any concrete example?
>>
>>
>> On 9 April 2014 08:29, Edward J. Yoon <edwardyoon@apache.org> wrote:
>>> In my eyes, SuperstepPiEstimator[1] look like totally new programming
>>> model, very similar with Pregel.
>>>
>>> I personally would like to suggest that we provide both pure BSP and
>>> fault tolerant BSP model, instead of replace.
>>>
>>> 1. http://svn.apache.org/repos/asf/hama/trunk/examples/src/main/java/org/apache/hama/examples/SuperstepPiEstimator.java
>>>
>>> --
>>> Edward J. Yoon (@eddieyoon)
>>> Chief Executive Officer
>>> DataSayer, Inc.
>
>
>
> --
> Edward J. Yoon (@eddieyoon)
> CEO at DataSayer Co., Ltd.

Mime
View raw message