hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1097) SequenceFile.Reader keeps around buffer whose size is that of largest item read -> results in lots of dead heap
Date Tue, 30 Dec 2008 15:28:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12659829#action_12659829
] 

Andrew Purtell commented on HBASE-1097:
---------------------------------------

+1 for commit.

As you might expect, heaps on our HRS are running 40-60% smaller than before. :-) Great work
Stack, bringing MapFile and SequenceFile local was a great move.

> SequenceFile.Reader keeps around buffer whose size is that of largest item read ->
results in lots of dead heap
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1097
>                 URL: https://issues.apache.org/jira/browse/HBASE-1097
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: apurtell 25 node TRUNK on hadoop 0.18.1 cluster
>            Reporter: stack
>             Fix For: 0.19.0
>
>         Attachments: mf.patch
>
>
> Andrew is OOMEing again.  Looking at some of his heaps, I can count Reader with DataOutputBuffers
of ~600MB in a 2G heap.  Testing I see that the DataOutputBuffer allocated at head of Mapfile.Reader
is reused when we call next, a reset is called. If I trace, the DataOutputBuffer has in it
an internal Buffer class which is based on ByteArrayOutputStream.  Reset of the DOB eventually
goes through to the BAOS reset.  This just sets the position.  It keeps the buffer sized to
whatever it grew to last time this BAOS was used (Figuring this was a little complicated by
the fact that DOB does some fancy footwork in a reset override to avoid copies).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message