kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewen Cheslack-Postava (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-2482) Allow copycat sink tasks to pause/resume consumption of specific topic partitions
Date Thu, 27 Aug 2015 17:10:46 GMT
Ewen Cheslack-Postava created KAFKA-2482:
--------------------------------------------

             Summary: Allow copycat sink tasks to pause/resume consumption of specific topic
partitions
                 Key: KAFKA-2482
                 URL: https://issues.apache.org/jira/browse/KAFKA-2482
             Project: Kafka
          Issue Type: Sub-task
          Components: copycat
            Reporter: Ewen Cheslack-Postava
            Assignee: Ewen Cheslack-Postava


Consider a situation where a sink is assigned 2 topic partitions. One of them runs into a
transient issue and no more data from it can be processed. However, the other topic partition
is proceeding fine. We don't want to block the second partition by constantly throwing exceptions
due to data from the first topic partition.

The new consumer now supports pause/resume, so we should expose these to the task. We could
expose the functionality directly, although that would also make the task responsible for
scheduling some task in the future to check whether it can resume. Another approach might
be to make the API include the backoff time. Then the framework would automatically resume
consumption of the topic partition after that time, which would presumably prompt the task
to reevaluate the situation for the topic partition when it receives another message for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message