kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Per Steffensen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5505) Connect: Do not restart connector and existing tasks on task-set change
Date Sun, 29 Oct 2017 15:52:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16224054#comment-16224054

Per Steffensen commented on KAFKA-5505:

[~gauravnanda] it sounds like your problem is slightly different than the one actually reported
here. I reported that it is annoying that all tasks and the connector itself is restarted
when an existing connector changes its set of tasks (by calling {{context.requestTaskReconfiguration()}}).
It seems like you are annoyed that all existing tasks and connectors are restarted when you
create a new connector. While the problems differ slightly, the solution is probably "kinda
the same", so maybe we should just make this ticket (KAFKA-5505) cover both?

> Connect: Do not restart connector and existing tasks on task-set change
> -----------------------------------------------------------------------
>                 Key: KAFKA-5505
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5505
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions:
>            Reporter: Per Steffensen
> I am writing a connector with a frequently changing task-set. It is really not working
very well, because the connector and all existing tasks are restarted when the set of tasks
changes. E.g. if the connector is running with 10 tasks, and an additional task is needed,
the connector itself and all 10 existing tasks are restarted, just to make the 11th task run
also. My tasks have a fairly heavy initialization, making it extra annoying. I would like
to see a change, introducing a "mode", where only new/deleted tasks are started/stopped when
notifying the system that the set of tasks changed (calling context.requestTaskReconfiguration()
- or something similar).
> Discussed this issue a little on dev@kafka.apache.org in the thread "Kafka Connect: To
much restarting with a SourceConnector with dynamic set of tasks"

This message was sent by Atlassian JIRA

View raw message