kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ismael Juma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-5512) KafkaConsumer: High memory allocation rate when idle
Date Mon, 26 Jun 2017 09:58:01 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-5512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ismael Juma updated KAFKA-5512:
-------------------------------
    Labels: performance  (was: )

> KafkaConsumer: High memory allocation rate when idle
> ----------------------------------------------------
>
>                 Key: KAFKA-5512
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5512
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.10.1.1
>            Reporter: Stephane Roset
>              Labels: performance
>             Fix For: 0.11.0.1
>
>
> Hi,
> We noticed in our application that the memory allocation rate increased significantly
when we have no Kafka messages to consume. We isolated the issue by using a JVM that simply
runs 128 Kafka consumers. These consumers consume 128 partitions (so each consumer consumes
one partition). The partitions are empty and no message has been sent during the test. The
consumers were configured with default values (session.timeout.ms=30000, fetch.max.wait.ms=500,
receive.buffer.bytes=65536, heartbeat.interval.ms=3000, max.poll.interval.ms=300000, max.poll.records=500).
The Kafka cluster was made of 3 brokers. Within this context, the allocation rate was about
55 MiB/s. This high allocation rate generates a lot of GC activity (to garbage the young heap)
and was an issue for our project.
> We profiled the JVM with JProfiler. We noticed that there were a huge quantity of ArrayList$Itr
in memory. These collections were mainly instantiated by the methods handleCompletedReceives,
handleCompletedSends, handleConnecions and handleDisconnections of the class NetWorkClient.
We also noticed that we had a lot of calls to the method pollOnce of the class KafkaConsumer.

> So we decided to run only one consumer and to profile the calls to the method pollOnce.
We noticed that regularly a huge number of calls is made to this method, up to 268000 calls
within 100ms. The pollOnce method calls the NetworkClient.handle* methods. These methods iterate
on collections (even if they are empty), so that explains the huge number of iterators in
memory.
> The large number of calls is related to the heartbeat mechanism. The pollOnce method
calculates the poll timeout; if a heartbeat needs to be done, the timeout will be set to 0.
The problem is that the heartbeat thread checks every 100 ms (default value of retry.backoff.ms)
if a heartbeat should be sent, so the KafkaConsumer will call the poll method in a loop without
timeout until the heartbeat thread awakes. For example: the heartbeat thread just started
to wait and will awake in 99ms. So during 99ms, the KafkaConsumer will call in a loop the
pollOnce method and will use a timeout of 0. That explains how we can have 268000 calls within
100ms. 
> The heartbeat thread calls the method AbstractCoordinator.wait() to sleep, so I think
the Kafka consumer should awake the heartbeat thread with a notify when needed.
> We made two quick fixes to solve this issue:
>   - In NetworkClient.handle*(), we don't iterate on collections if they are empty (to
avoid unnecessary iterators instantiations).
>   - In KafkaConsumer.pollOnce(), if the poll timeout is equal to 0 we notify the heartbeat
thread to awake it (dirty fix because we don't handle the autocommit case).
> With these 2 quick fixes and 128 consumers, the allocation rate drops down from 55 MiB/s
to 4 MiB/s.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message