Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1569759164.1228862084301.JavaMail.jira@brutus>
Date: Tue, 9 Dec 2008 14:34:44 -0800 (PST)
From: "stack (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4802) RPC Server send buffer retains size
 of largest response ever sent
In-Reply-To: <828234010.1228679144140.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654990#action_12654990 ] 

stack commented on HADOOP-4802:
-------------------------------

> I'm still not convinced we should do more than replace the buf.reset() with buf = new ByteArrayOutputStream() and remove the initialization of buf altogether.

Above is predicated on our running a 'benchmark'.  What would you suggest I run?

Why create new objects when it can be avoided (when response < 10k)?


> RPC Server send buffer retains size of largest response ever sent 
> ------------------------------------------------------------------
>
>                 Key: HADOOP-4802
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4802
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.18.2, 0.19.0
>            Reporter: stack
>         Attachments: 4802-v2.patch, 4802.patch
>
>
> The stack-based ByteArrayOutputStream in Server.Hander is reset each time through the run loop.  This will set the BAOS 'size' back to zero but the allocated backing buffer is unaltered.  If during an Handlers' lifecycle, any particular RPC response was fat -- Megabytes, even -- the buffer expands during the write to accommodate the particular response but then never shrinks subsequently.  If a hosting Server has had more than one 'fat payload' occurrence, the resultant occupied heap can provoke memory woes (See https://issues.apache.org/jira/browse/HBASE-900?focusedCommentId=12654009#action_12654009 for an extreme example; occasional payloads of 20-50MB with 30 handlers robbed the heap of 700MB).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.