hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wangchao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-12619) Native memory leaks in CompressorStream
Date Mon, 07 Dec 2015 15:14:10 GMT
wangchao created HADOOP-12619:

             Summary: Native memory leaks in CompressorStream
                 Key: HADOOP-12619
                 URL: https://issues.apache.org/jira/browse/HADOOP-12619
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: 2.4.0
            Reporter: wangchao

The constructor of org.apache.hadoop.io.compress.CompressorStream requires an org.apache.hadoop.io.compress.Compressor
 object to compress bytes but it does not invoke the compressor's finish method when close
method are called. This may causes the native memory leaks if the compressor is only used
by this CompressorStream object.

I found this when set up a flume agent with gzip compression, the native memory grows slowly
and cannot fall back. 

  public CompressionOutputStream createOutputStream(OutputStream out) 
    throws IOException {
    return (ZlibFactory.isNativeZlibLoaded(conf)) ?
               new CompressorStream(out, createCompressor(),
                                    conf.getInt("io.file.buffer.size", 4*1024)) :
               new GzipOutputStream(out);

  public Compressor createCompressor() {
    return (ZlibFactory.isNativeZlibLoaded(conf))
      ? new GzipZlibCompressor(conf)
      : null;

The method of CompressorStream is

  public void close() throws IOException {
    if (!closed) {
      closed = true;

  public void finish() throws IOException {
    if (!compressor.finished()) {
      while (!compressor.finished()) {

No one will end the compressor.

This message was sent by Atlassian JIRA

View raw message