hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Marion (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12619) Native memory leaks in CompressorStream
Date Wed, 21 Dec 2016 19:14:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15767893#comment-15767893
] 

Dave Marion commented on HADOOP-12619:
--------------------------------------

Related to HADOOP-12007?

> Native memory leaks in CompressorStream
> ---------------------------------------
>
>                 Key: HADOOP-12619
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12619
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.4.0
>            Reporter: wangchao
>
> The constructor of org.apache.hadoop.io.compress.CompressorStream requires an org.apache.hadoop.io.compress.Compressor
 object to compress bytes but it does not invoke the compressor's finish method when close
method are called. This may causes the native memory leaks if the compressor is only used
by this CompressorStream object.
> I found this when set up a flume agent with gzip compression, the native memory grows
slowly and cannot fall back. 
> {code}
>   @Override
>   public CompressionOutputStream createOutputStream(OutputStream out) 
>     throws IOException {
>     return (ZlibFactory.isNativeZlibLoaded(conf)) ?
>                new CompressorStream(out, createCompressor(),
>                                     conf.getInt("io.file.buffer.size", 4*1024)) :
>                new GzipOutputStream(out);
>   }
>   @Override
>   public Compressor createCompressor() {
>     return (ZlibFactory.isNativeZlibLoaded(conf))
>       ? new GzipZlibCompressor(conf)
>       : null;
>   }
> {code}
> The method of CompressorStream is
> {code}
>   @Override
>   public void close() throws IOException {
>     if (!closed) {
>       finish();
>       out.close();
>       closed = true;
>     }
>   }
>   @Override
>   public void finish() throws IOException {
>     if (!compressor.finished()) {
>       compressor.finish();
>       while (!compressor.finished()) {
>         compress();
>       }
>     }
>   }
> {code}
> No one will end the compressor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message