kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3937) Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages
Date Tue, 16 Aug 2016 14:36:20 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422805#comment-15422805
] 

ASF GitHub Bot commented on KAFKA-3937:
---------------------------------------

GitHub user wiyu opened a pull request:

    https://github.com/apache/kafka/pull/1743

    KAFKA-3937: Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

    @ijuma - Creating this PR against 0.9.0 as this is what we're using in prod. I can modify
this for trunk if the logic solution looks good. 
    
    I modified the record iterator to inherit from closable and implemented the method to
call streams.close(). The stream will now be closed when we are done traversing the inner
iterator. I had to move the `try {} catch{}` out of the `if` portion to deal with IOException
coming from `innerDone()`.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/wiyu/kafka compressor_memory_leak_in_fetcher

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/1743.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1743
    
----
commit b56ec1563e34d40c338224970c6b8cd3802e4dcd
Author: William Yu <wyu@unified.com>
Date:   2016-08-16T14:09:28Z

    KAFKA-3937: Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

----


> Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages
> --------------------------------------------------------------------------------
>
>                 Key: KAFKA-3937
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3937
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions: 0.8.2.2, 0.9.0.1, 0.10.0.0
>         Environment: Linux, latest oracle java-8
>            Reporter: Tom Crayford
>            Priority: Minor
>
> In https://issues.apache.org/jira/browse/KAFKA-3933, we discovered that brokers can crash
when performing log recovery, as they leak native memory whilst decompressing compressed segments,
and that native memory isn't cleaned up rapidly enough by garbage collection and finalizers.
The work to fix that in the brokers is taking part in https://github.com/apache/kafka/pull/1598.
As part of that PR, Ismael Juma asked me to fix similar issues in the client. Rather than
have one large PR that fixes everything, I'd rather break this work up into seperate things,
so I'm filing this JIRA to track the followup work. I should get to a PR on this at some point
relatively soon, once the other PR has landed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message