tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-3140) Reduce AM memory usage while serialization
Date Fri, 26 Feb 2016 21:49:18 GMT

    [ https://issues.apache.org/jira/browse/TEZ-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15169901#comment-15169901
] 

Jason Lowe commented on TEZ-3140:
---------------------------------

There are other cases in Hadoop code where they need to do something similar, and if we want
to avoid the custom ByteBufferDataInput class I think we could reuse the DataInputByteBuffer
and accomplish the same no-copy semantics like this:
{code}
        if (out instanceof DataOutputBuffer) {
          DataOutputBuffer dub = (DataOutputBuffer) out;
          DataInputByteBuffer dibb = new DataInputByteBuffer();
          dibb.reset(bb);
          dub.write(dibb, size);
{code}

ShuffleUtils and DagTypeConverters are already doing similar things with DataInputByteBuffer.


> Reduce AM memory usage while serialization
> ------------------------------------------
>
>                 Key: TEZ-3140
>                 URL: https://issues.apache.org/jira/browse/TEZ-3140
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.7.1, 0.8.3
>
>         Attachments: TEZ-3140-1.patch
>
>
>    There is an unnecessary copy of userpayload byte array during serialization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message