kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manikumar Reddy (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException
Date Wed, 17 Aug 2016 11:23:20 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15424315#comment-15424315

Manikumar Reddy edited comment on KAFKA-3980 at 8/17/16 11:22 AM:

as suspected, this may be related to quota metrics leak : KAFKA-4054

cc [~rsivaram]

was (Author: omkreddy):
as suspected, this may be related to quota leak : KAFKA-4054

cc [~rsivaram]

> JmxReporter uses excessive memory causing OutOfMemoryException
> --------------------------------------------------------------
>                 Key: KAFKA-3980
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3980
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions:
>            Reporter: Andrew Jorgensen
> I have some nodes in a kafka cluster that occasionally will run out of memory whenever
I restart the producers. I was able to take a heap dump from both a recently restarted Kafka
node which weighed in at about 20 MB and a node that has been running for 2 months is using
over 700MB of memory. Looking at the heap dump it looks like the JmxReporter is holding on
to metrics and causing them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the producers
it will cause the node to experience an Java heap space exception and OOM. The nodes  then
fail to startup correctly and write a -1 as the leader number to the partitions they were
responsible for effectively resetting the offset and rendering that partition unavailable.
The kafka process then needs to go be restarted in order to re-assign the node to the partition
that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount of time
or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.

This message was sent by Atlassian JIRA

View raw message