avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Bowzer (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AVRO-1896) Leaking Deflater Objects
Date Fri, 05 Aug 2016 04:32:20 GMT
Chris Bowzer created AVRO-1896:

             Summary: Leaking Deflater Objects
                 Key: AVRO-1896
                 URL: https://issues.apache.org/jira/browse/AVRO-1896
             Project: Avro
          Issue Type: Improvement
          Components: java
            Reporter: Chris Bowzer
            Priority: Minor

In the java version, with normal creation and use of a DeflateCodec there is a reference to
a Deflater object that never has the end method called on it.  While this item still eventually
gets garbage collected and release it's c-heap resources, I have run into the issue of a build
up of these objects (and the resources they are holding) when using many instances of these
objects in parallel.  This has caused a hard to diagnose memory issue where the java heap
keeps getting smaller while all resources we have access to are being closed.
While we have solved this issued by recycling DataFileWriters so that there is only ever a
limited amount of DeflateCodec instances to be gc'd, this is not an ideal solution. It would
be great if there was some way of closing the DeflateCodec instance held inside a DataFileWriter
so that deflater.end() can be called explicitly rather than waiting for the finalizer to run.

This message was sent by Atlassian JIRA

View raw message