reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Weimer (JIRA)" <j...@apache.org>
Subject [jira] [Created] (REEF-1157) Improve memory usage and performance of the Java group communications code
Date Tue, 26 Jan 2016 20:38:40 GMT
Markus Weimer created REEF-1157:
-----------------------------------

             Summary: Improve memory usage and performance of the Java group communications
code
                 Key: REEF-1157
                 URL: https://issues.apache.org/jira/browse/REEF-1157
             Project: REEF
          Issue Type: Improvement
          Components: GroupCommunications
            Reporter: Markus Weimer
            Priority: Minor


The Group Communications library in .NET has seen two areas of improvement that we should
follow up on in the Java code:

  * Zero copies of the data being sent: the current code takes the user data, serializes it
and then wraps it with the operator's envelope. That package is serialized and wrapped in
the group communications envelope. Again, this is serialized and wrapped in the network service's
envelope. That is what is eventually serialized and sent. All of these wrappings and serializations
create copies of the (potentially large) user data. We should move to a zero-copy regime where
the data is only serialized once.
  * Pipelining: Many times, group communications is applied with vector types, e.g. when summing
up {{double[]}}. Those operations can be done on a per-dimension basis, which reduces the
memory load and improves throughput.

This JIRA shall serve as an anchor for work done in this area.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message