hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Phabricator (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-5387) Reuse compression streams in HFileBlock.Writer
Date Sun, 12 Feb 2012 01:55:00 GMT

     [ https://issues.apache.org/jira/browse/HBASE-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Phabricator updated HBASE-5387:
-------------------------------

    Attachment: D1719.3.patch

mbautin updated the revision "[jira] [HBASE-5387] Reuse compression streams in HFileBlock.Writer".
Reviewers: tedyu, Liyin, dhruba, JIRA

  Breaking compression stream creation methods into multiple parts. Avoiding using "finish-on-flush"
stream in a couple of places, and deprecating the currently used method that is always called
with downstream buffer size of 0, at least within HBase code. The method is still there for
backwards compatibility. Also, getting rid of hard-coded buffer size by delegating the native
compressor case to superclass in ReusableStreamGzipCodec.

REVISION DETAIL
  https://reviews.facebook.net/D1719

AFFECTED FILES
  src/main/java/org/apache/hadoop/hbase/io/hfile/Compression.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV1.java
  src/main/java/org/apache/hadoop/hbase/io/hfile/ReusableStreamGzipCodec.java
  src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java
  src/test/java/org/apache/hadoop/hbase/regionserver/DataBlockEncodingTool.java

                
> Reuse compression streams in HFileBlock.Writer
> ----------------------------------------------
>
>                 Key: HBASE-5387
>                 URL: https://issues.apache.org/jira/browse/HBASE-5387
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.94.0
>            Reporter: Mikhail Bautin
>            Assignee: Mikhail Bautin
>            Priority: Critical
>             Fix For: 0.94.0
>
>         Attachments: D1719.1.patch, D1719.2.patch, D1719.3.patch, Fix-deflater-leak-2012-02-10_18_48_45.patch,
Fix-deflater-leak-2012-02-11_17_13_10.patch
>
>
> We need to to reuse compression streams in HFileBlock.Writer instead of allocating them
every time. The motivation is that when using Java's built-in implementation of Gzip, we allocate
a new GZIPOutputStream object and an associated native data structure every time we create
a compression stream. The native data structure is only deallocated in the finalizer. This
is one suspected cause of recent TestHFileBlock failures on Hadoop QA: https://builds.apache.org/job/HBase-TRUNK/2658/testReport/org.apache.hadoop.hbase.io.hfile/TestHFileBlock/testPreviousOffset_1_/.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message