tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Seth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-3140) Reduce AM memory usage while serialization
Date Fri, 26 Feb 2016 07:15:18 GMT

    [ https://issues.apache.org/jira/browse/TEZ-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15168581#comment-15168581
] 

Siddharth Seth commented on TEZ-3140:
-------------------------------------

The main intent here is to avoid creating huge buffers while sending data to tasks - use the
buffer created by the RPC layer itself, otherwise restrict to 8K at max ?
We really should move this over to the Protobuf engine - though that may end up forcing the
buffer creation as well.

The changes look good to me.
Minor: SERIALIZE_BUFFER_SIZE, ByteBufferDataInput.bb should be final
I think it's worthwhile to add a unit test for the serialization loop - a simple test to write
an EntityDescriptor with a payload less than 8K and another > 8K. ByteBuffer changes have
caused problems in the past.

> Reduce AM memory usage while serialization
> ------------------------------------------
>
>                 Key: TEZ-3140
>                 URL: https://issues.apache.org/jira/browse/TEZ-3140
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.7.1, 0.8.3
>
>         Attachments: TEZ-3140-1.patch
>
>
>    There is an unnecessary copy of userpayload byte array during serialization.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message