reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "amil shanaka (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (REEF-1157) Improve memory usage and performance of the Java group communications code
Date Sat, 05 Mar 2016 04:17:40 GMT

    [ https://issues.apache.org/jira/browse/REEF-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181500#comment-15181500
] 

amil shanaka edited comment on REEF-1157 at 3/5/16 4:17 AM:
------------------------------------------------------------

Hi,
I'm a 3rd year under graduate in the department of Computer Science and Engineering at University
of Moratuwa,Sri Lanka. In my experience, I've been worked with Java nio.ByteBuffer and also
off-heap memory usage like java Unsafe,  nio.directByteBuffer and netty Direct Byte Buf for
enhancing the performance with communicating meta data. So i would like to know more details
regrading this project in detail that I could proceed .
Thank you,
amil


was (Author: amil101):
Hi,
I'm a 3rd year under graduate in the department of Computer Science and Engineering at University
of Moratuwa,Sri Lanka. In my experience, I've been worked with Java nio.ByteBuffer and also
off-heap memory usage like nio.directByteBuffer and netty Direct Byte Buf for enhancing the
performance with communicating meta data. So i would like to know more details regrading this
project in detail that I could proceed .
Thank you,
amil

> Improve memory usage and performance of the Java group communications code
> --------------------------------------------------------------------------
>
>                 Key: REEF-1157
>                 URL: https://issues.apache.org/jira/browse/REEF-1157
>             Project: REEF
>          Issue Type: Improvement
>          Components: GroupCommunications
>            Reporter: Markus Weimer
>            Priority: Minor
>              Labels: GSOC, gsoc2016, java
>
> The Group Communications library in .NET has seen two areas of improvement that we should
follow up on in the Java code:
>   * Zero copies of the data being sent: the current code takes the user data, serializes
it and then wraps it with the operator's envelope. That package is serialized and wrapped
in the group communications envelope. Again, this is serialized and wrapped in the network
service's envelope. That is what is eventually serialized and sent. All of these wrappings
and serializations create copies of the (potentially large) user data. We should move to a
zero-copy regime where the data is only serialized once.
>   * Pipelining: Many times, group communications is applied with vector types, e.g. when
summing up {{double[]}}. Those operations can be done on a per-dimension basis, which reduces
the memory load and improves throughput.
> This JIRA shall serve as an anchor for work done in this area.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message