kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Per Steffensen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
Date Fri, 23 Jun 2017 08:08:00 GMT
Per Steffensen created KAFKA-5505:
-------------------------------------

             Summary: Connect: Do not restart connector and existing tasks on task-set change
                 Key: KAFKA-5505
                 URL: https://issues.apache.org/jira/browse/KAFKA-5505
             Project: Kafka
          Issue Type: Improvement
          Components: KafkaConnect
    Affects Versions: 0.10.2.1
            Reporter: Per Steffensen


I am writing a connector with a frequently changing task-set. It is really not working very
well, because the connector and all existing tasks are restarted when the set of tasks changes.
E.g. if the connector is running with 10 tasks, and an additional task is needed, the connector
itself and all 10 existing tasks are restarted, just to make the 11th task run also. My tasks
have a fairly heavy initialization, making it extra annoying. I would like to see a change,
introducing a "mode", where only new/deleted tasks are started/stopped when notifying the
system that the set of tasks changed (calling context.requestTaskReconfiguration() - or something
similar).

Discussed this issue a little on dev@kafka.apache.org in the thread "Kafka Connect: To much
restarting with a SourceConnector with dynamic set of tasks"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message