flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "JiYeon OH (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLUME-3086) KafkaChannel commits consume offset to Zookeeper every time causing performance problem
Date Fri, 14 Apr 2017 08:00:52 GMT
JiYeon OH created FLUME-3086:
--------------------------------

             Summary: KafkaChannel commits consume offset to Zookeeper every time causing
performance problem
                 Key: FLUME-3086
                 URL: https://issues.apache.org/jira/browse/FLUME-3086
             Project: Flume
          Issue Type: Bug
          Components: Kafka Channel
    Affects Versions: 1.7.0
            Reporter: JiYeon OH


Currently, the AUTO_COMMIT_ENABLED setting is hard corded to false(KafkaChannel.java line
285), and the commit is done every transaction(KafkaChannel.java line 567). This will cause
large overhead to Kafka Zookeeper when consuming very large amount of data because every consumer
in every node will be writing to Zookeeper. 
I actually had this problem and affected other running consumers' offsets on the Zookeeper.
So I suggest to change the hard coded setting AUTO_COMMIT_ENABLED to true, and set a default
AUTO_COMMIT_INTERVAL value, and remove the commitOffsets() method. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message