Mailing-List: contact jira-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@kafka.apache.org
Date: Fri, 27 Oct 2017 21:10:01 +0000 (UTC)
From: "Gaurav Nanda (JIRA)" <jira@apache.org>
To: jira@kafka.apache.org
Message-ID: <JIRA.13082053.1498205253000.82692.1509138601547@Atlassian.JIRA>
In-Reply-To: <JIRA.13082053.1498205253000@Atlassian.JIRA>
References: <JIRA.13082053.1498205253000@Atlassian.JIRA> <JIRA.13082053.1498205253957@jira-lw-us.apache.org>
Subject: [jira] [Commented] (KAFKA-5505) Connect: Do not restart connector
 and existing tasks on task-set change
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 27 Oct 2017 21:10:08 -0000


    [ https://issues.apache.org/jira/browse/KAFKA-5505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222835#comment-16222835 ] 

Gaurav Nanda commented on KAFKA-5505:
-------------------------------------

We also have been facing this issue, we have a bunch of source connectors running on a connect cluster in production, everytime we create a new connector all the existing connectors are restarted for rebalancing. This disrupts the data loading process for some time, more then number of connectors and tasks, longer it takes for the process to get back up and running. 

We should be able to avoid restarting all existing connectors if a new one is added to the cluster. 

The only solution for us is to have a new cluster created dynamically when we need a new connector. Which is a clunky solution ...
I have talked to some people in meetups and have found more people who are affected by this. I would really recommend fixing this.

> Connect: Do not restart connector and existing tasks on task-set change
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-5505
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5505
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 0.10.2.1
>            Reporter: Per Steffensen
>
> I am writing a connector with a frequently changing task-set. It is really not working very well, because the connector and all existing tasks are restarted when the set of tasks changes. E.g. if the connector is running with 10 tasks, and an additional task is needed, the connector itself and all 10 existing tasks are restarted, just to make the 11th task run also. My tasks have a fairly heavy initialization, making it extra annoying. I would like to see a change, introducing a "mode", where only new/deleted tasks are started/stopped when notifying the system that the set of tasks changed (calling context.requestTaskReconfiguration() - or something similar).
> Discussed this issue a little on dev@kafka.apache.org in the thread "Kafka Connect: To much restarting with a SourceConnector with dynamic set of tasks"


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)