hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apurv Verma (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-515) Counter should be synchronized by sync()
Date Sun, 04 Mar 2012 14:12:00 GMT

    [ https://issues.apache.org/jira/browse/HAMA-515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221896#comment-13221896
] 

Apurv Verma commented on HAMA-515:
----------------------------------

Hii,
 Here is a solution I have come up with.
 Basically we maintain two counter values. One for the value of the counter in the current
superstep and the other for the value of the counter at the end of the last superstep.

{noformat}
  @Override
  public final void sync() throws IOException, SyncException,
      InterruptedException {
    enterBarrier();
    Iterator<Entry<InetSocketAddress, LinkedList<M>>> it = messenger
        .getMessageIterator();

    while (it.hasNext()) {
      Entry<InetSocketAddress, LinkedList<M>> entry = it.next();
      final InetSocketAddress addr = entry.getKey();
      final Iterable<M> messages = entry.getValue();

      final BSPMessageBundle<M> bundle = combineMessages(messages);

      if (conf.getBoolean("bsp.checkpoint.enabled", false)) {
        checkpoint(checkpointedPath(), bundle);
      }

      // remove this message during runtime to save a bit of memory
      it.remove();

      messenger.transfer(addr, bundle);
    }



    lock();
    data = read(); //read the data from znode.
    write(currentCounterValue - lastCounterValue) + data;
    unlock();

    leaveBarrier();

    lastCounterVal    = currentCounterVal; 
    currentCounterVal = read();


    incrCounter(PeerCounter.SUPERSTEP_SUM, 1L);

    currentTaskStatus.setCounters(counters);

    umbilical.statusUpdate(taskId, currentTaskStatus);
    // Clear outgoing queues.
    messenger.clearOutgoingQueues();
  }
{noformat}


Thomas told me that sync() already has a lock so I don't have to worry about the lock part
of it. As far as my understanding about barriers goes it allows multiple threads (in our case
bsp tasks) to enter and only when all have arrived it allows them to leave.
I still don't see where they acquire the lock. I still have to see that up.

http://pastebin.com/XHHm0q1X
                
> Counter should be synchronized by sync()
> ----------------------------------------
>
>                 Key: HAMA-515
>                 URL: https://issues.apache.org/jira/browse/HAMA-515
>             Project: Hama
>          Issue Type: New Feature
>    Affects Versions: 0.4.0
>            Reporter: Thomas Jungblut
>            Priority: Minor
>             Fix For: 0.5.0
>
>
> We should synchronize the counters in all tasks after a sync() call. 
> Then someone can use it for flow control. E.G. graph related algorithms, to get the number
of globally updated vertices.
> Two options to solve this:
>  - Sync the whole stuff over RPC from the BSPMaster
>  - Use Zookeeper for counter handling

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message