hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suraj Menon <surajsme...@apache.org>
Subject Re: outgoingQueue and messages bundling
Date Wed, 15 Jan 2014 10:50:44 GMT
Nice! So you are reaching at some pain points of messaging code :). I
think, the code has been since I have seen it. It works for low number of
messages. For your concerns, there are message bundles not coming in the
path of the code yet in org.apache.hama.bsp.message.bundle . There is a
ByteBufferBSPMessageBundle in it. BSPMessageBundle represents unit piece of
data that could be serialized by sender and deserialized by receiver. It is
worthy for asynchronous communication. Your suggestion does make sense and
the whole thing requires code refactoring.

-Suraj



On Wed, Jan 15, 2014 at 3:44 AM, Edward J. Yoon <edwardyoon@apache.org>wrote:

> If we use Spilling or Disk queues for outgoing queue, some memory
> consumption can be saved but we can't avoid deserializing and bundling
> messages overhead. According to my test, disk-based outgoing queue
> doesn't help much in memory usage.
>
> So, in my opinion, we should add outgoing messages in bundle object
> directly, and should consider about byte array bundle, or disk-based
> bundle. We can add something like OutgoingMessageManager.
>
> Does it make sense?
>
> On Wed, Jan 15, 2014 at 2:32 PM, Edward J. Yoon <edwardyoon@apache.org>
> wrote:
> > As I described here,
> >
> http://wiki.apache.org/hama/Architecture#Communication_and_Synchronization_Process
> >
> > Currently, we puts the messages to be sent into outgoing queue. Then
> > we puts the messages of outgoing queue into BSPMessageBundle object
> > again.
> >
> > This is huge overhead, and very inefficient in memory usage. Why
> > should we do like that?
> >
> > Please let me know if i missing something important.
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message