hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MaoYuan Xian (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HAMA-759) When peer sends message, not matter what kinds of queue used, all messages read into memory.
Date Thu, 30 May 2013 11:08:19 GMT
MaoYuan Xian created HAMA-759:
---------------------------------

             Summary: When peer sends message, not matter what kinds of queue used, all messages
read into memory.
                 Key: HAMA-759
                 URL: https://issues.apache.org/jira/browse/HAMA-759
             Project: Hama
          Issue Type: Improvement
          Components: bsp core
    Affects Versions: 0.6.1
            Reporter: MaoYuan Xian


Before bsp peer sends message to network, all messages have to be read into memory, No matter
what kinds of outgoing queue used. Which means, even when user use DiskQueue to sending message,
no memory saving exactly. In sync() method of BSPPeerImpl.java, 

{code}
   while (it.hasNext()) {
      ... 
      BSPMessageBundle<M> bundle = combineMessages(messages);
      // remove this message during runtime to save a bit of memory
      it.remove();
      if (combiner != null) {
        try {
          messenger.transfer(addr, bundle);
        } catch (Exception e) {
          LOG.error("Error while sending messages", e);
        }
{code}

In the combineMessages method, all messages are put into memory:

{code}
  private final BSPMessageBundle<M> combineMessages(Iterable<M> messages) {
    BSPMessageBundle<M> bundle = new BSPMessageBundle<M>();
    if (combiner != null) {
      bundle.addMessage(combiner.combine(messages));
    } else {
      for (M message : messages) {
        bundle.addMessage(message);
      }
    }
    return bundle;
  }
{code}

Can we make some modifications here, implements a configurable size sending bundle, only collect
parts of message instead of all before sends messages to network.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message