kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gunnar Morling (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6551) Unbounded queues in WorkerSourceTask cause OutOfMemoryError
Date Fri, 09 Feb 2018 11:54:00 GMT
Gunnar Morling created KAFKA-6551:

             Summary: Unbounded queues in WorkerSourceTask cause OutOfMemoryError
                 Key: KAFKA-6551
                 URL: https://issues.apache.org/jira/browse/KAFKA-6551
             Project: Kafka
          Issue Type: Bug
          Components: KafkaConnect
            Reporter: Gunnar Morling

A Debezium user reported an {{OutOfMemoryError}} to us, with over 50,000 messages in the {{WorkerSourceTask#outstandingMessages}}

This map is unbounded and I can't see any way of "rate limiting" which would control how many
records are added to it. Growth can only indirectly be limited by reducing the offset flush
interval, but as connectors can return large amounts of messages in single {{poll()}} calls
that's not sufficient in all cases. Note the user reported this issue during snapshotting
a database, i.e. a high number of records arrived in a very short period of time.

To solve the problem I'd suggest to make this map backpressure-aware and thus prevent its
indefinite growth, so that no further records will be polled from the connector until messages
have been taken out of the map again.

This message was sent by Atlassian JIRA

View raw message