kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Noah Sloan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3892) Clients retain metadata for non-subscribed topics
Date Wed, 22 Jun 2016 20:27:16 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345103#comment-15345103

Noah Sloan commented on KAFKA-3892:

I believe the ultimate cause is that the either a request is being made for all topic metadata,
or a broker is mistakenly responding with all topic metadata. I was not able to figure out
why that would happen.

I think the best immediate fix is to have the client defensively prune the metadata response
and only retain topics that are subscribed. My PR does not affect the case where a topic pattern
is used for subscription, so it seems like a fairly safe change to me.

> Clients retain metadata for non-subscribed topics
> -------------------------------------------------
>                 Key: KAFKA-3892
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3892
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients
>    Affects Versions:
>            Reporter: Noah Sloan
> After upgrading to from 0.8.2 (and adopting the new consumer and producer classes,)
we noticed services with small heap crashing due to OutOfMemoryErrors. These services contained
many producers and consumers (~20 total) and were connected to brokers with >2000 topics
and over 10k partitions. Heap dumps revealed that each client had 3.3MB of Metadata retained
in their Cluster, with references to topics that were not being produced or subscribed to.
While the services were running with 128MB of heap prior to the upgrade, we to had increased
max heap to 200MB to accommodate all the extra data. 
> While this is not technically a memory leak, it does impose a significant overhead on
clients when connected to a large cluster.

This message was sent by Atlassian JIRA

View raw message