kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "William Yu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3937) Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages
Date Fri, 12 Aug 2016 02:42:20 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15418276#comment-15418276
] 

William Yu commented on KAFKA-3937:
-----------------------------------

You're referring to the Kafka consumer when you say 'kafka client' correct? I believe this
is the bug ticket I was looking for to confirm the behavior that I'm seeing from my consumer
application. It looks to be leaking native memory while consuming from a topic as my java
-Xmx setting is set for 10gb, but the resident memory is showing 23gb usage.

> Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages
> --------------------------------------------------------------------------------
>
>                 Key: KAFKA-3937
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3937
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions: 0.8.2.2, 0.9.0.1, 0.10.0.0
>         Environment: Linux, latest oracle java-8
>            Reporter: Tom Crayford
>            Priority: Minor
>
> In https://issues.apache.org/jira/browse/KAFKA-3933, we discovered that brokers can crash
when performing log recovery, as they leak native memory whilst decompressing compressed segments,
and that native memory isn't cleaned up rapidly enough by garbage collection and finalizers.
The work to fix that in the brokers is taking part in https://github.com/apache/kafka/pull/1598.
As part of that PR, Ismael Juma asked me to fix similar issues in the client. Rather than
have one large PR that fixes everything, I'd rather break this work up into seperate things,
so I'm filing this JIRA to track the followup work. I should get to a PR on this at some point
relatively soon, once the other PR has landed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message