hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhihong Yu (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
Date Sat, 11 Feb 2012 22:32:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13206287#comment-13206287
] 

Zhihong Yu edited comment on HBASE-5387 at 2/11/12 10:32 PM:
-------------------------------------------------------------

We could also do something heuristically. Get a random float and if it is < 0.001 (for
example, every 1000's time on average) throw away the old BAOS and get a new one.
Can the BAOS really get > the blocksize?

Also what happens when somebody (foolishly, but it does happen) sets the block size to 512mb,
in that case we might want to free this BOAS every single time.
                
      was (Author: lhofhansl):
    We could also do something heuristically. Get a random float and if it is < 0.001 (for
example, every 1000's time on average) throw away the old BOAS and get a new one.
Can the BOAS really get > the the blocksize?

Also what happens when somebody (foolishly, but it does happen) sets the block size to 512mb,
in that case we might want to free this BOAS every single time.
                  
> Reuse compression streams in HFileBlock.Writer
> ----------------------------------------------
>
>                 Key: HBASE-5387
>                 URL: https://issues.apache.org/jira/browse/HBASE-5387
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.94.0
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Critical
>             Fix For: 0.94.0
>
>         Attachments: D1719.1.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch
>
>
> We need to to reuse compression streams in HFileBlock.Writer instead of allocating them
every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate
a new GZIPOutputStream object and an associated native data structure every time we create
a compression stream. The native data structure is only deallocated in the finalizer. This
is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message