curator-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CURATOR-487) GzipCompressionProvider produces a lot of finalizable objects
Date Mon, 10 Dec 2018 15:24:00 GMT

    [ https://issues.apache.org/jira/browse/CURATOR-487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714909#comment-16714909
] 

ASF GitHub Bot commented on CURATOR-487:
----------------------------------------

Github user leventov commented on the issue:

    https://github.com/apache/curator/pull/282
  
    There is no peer evidence here, because we are on the optimization forefront. See https://github.com/apache/incubator-druid/pull/6677#discussion_r237182258
and https://lists.apache.org/thread.html/1aff123193cec5c385821b2d745a4e846a8a5786146c047acbdf8ea3@%3Cdev.druid.apache.org%3E.
    
    I've seen a Druid heap with more than 10k finalizable Deflater objects, about 8k of which
were already dead, awaiting in the finalization queue. They come from `GzipCompressionProvider`.
    
    Historically Druid uses Zookeeper somewhat wrong (not for what Zookeeper was designed):
it announces data segment placement using Zookeeper, that leads to creation of a lot of new
nodes in Zookeeper every second. It means that by accident, Druid is a good stress test for
Zookeeper (and consequently for Curator), and we run probably the largest Druid cluster.


> GzipCompressionProvider produces a lot of finalizable objects
> -------------------------------------------------------------
>
>                 Key: CURATOR-487
>                 URL: https://issues.apache.org/jira/browse/CURATOR-487
>             Project: Apache Curator
>          Issue Type: Improvement
>            Reporter: Roman Leventov
>            Assignee: Cameron McKenzie
>            Priority: Major
>             Fix For: 4.1.0
>
>
> GzipCompressionProvider.compress() and decompress() methods are called a lot inside and
outside of the framework and each such call produces a java.util.zip.Deflater or Inflater
object that are finalizable, that is bad for GC.
>  
> Compressing or decompressing a finite byte[] object (in contrast to compressing/decompressing
an InputStream or an OutputStream of unknown length) is actually a happy case because even
PhantomReference object could be avoided, the native resources could be created and freed
in a single try-finally block.
>  
> Curator must avoid that. [https://github.com/ymnk/jzlib] is a potential replacement,
for example.
>  
> A similar issue could be also fixed in Apache Commons Compress: https://issues.apache.org/jira/browse/COMPRESS-473 and
Curator is made to depend on Commons Compress.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message