kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajini Sivaram (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-6854) Log cleaner fails with transaction markers that are deleted during clean
Date Thu, 03 May 2018 12:23:00 GMT
Rajini Sivaram created KAFKA-6854:
-------------------------------------

             Summary: Log cleaner fails with transaction markers that are deleted during clean
                 Key: KAFKA-6854
                 URL: https://issues.apache.org/jira/browse/KAFKA-6854
             Project: Kafka
          Issue Type: Task
          Components: core
    Affects Versions: 1.1.0
            Reporter: Rajini Sivaram
            Assignee: Rajini Sivaram


Log cleaner grows buffers when `result.messagesRead` is zero. In a typical scenario, this
is a result of source buffer being too small to read the first batch. The buffer is then doubled
in size until one batch can be read, up to a maximum of `max.message.size`. There are issues
with the maximum message size used in calculations as reported in KAFKA-6834. But there is
a separate issue with the use of `result.messagesRead` when transactions are used. This contains
the number of filtered messages read from source which can be zero when a transaction control
marker is discarded. Log cleaner incorrectly assumes that messages were not read because the
buffer was too small. This can result in the log cleaner attempting to grow buffers forever,
failing with an exception when `max.message.bytes` is reached. This kills the log cleaner.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message