kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Satyajit varma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6208) Reduce startup time for Kafka Connect workers
Date Fri, 09 Feb 2018 00:19:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357745#comment-16357745
] 

Satyajit varma commented on KAFKA-6208:
---------------------------------------

[~rhauch] , we could make the connectors,SMT's and converters load asynchronously, like below,

{code}

CompletableFuture.runAsync(()->{
 addPlugins(plugins.connectors(), loader);
 connectors.addAll(plugins.connectors());
}, Executors.newSingleThreadExecutor());


CompletableFuture.runAsync(()->{
 addPlugins(plugins.converters(), loader);
 converters.addAll(plugins.converters());
}, Executors.newSingleThreadExecutor());


CompletableFuture.runAsync(()->{
 addPlugins(plugins.transformations(), loader);
 transformations.addAll(plugins.transformations());
}, Executors.newSingleThreadExecutor());

{code}

> Reduce startup time for Kafka Connect workers
> ---------------------------------------------
>
>                 Key: KAFKA-6208
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6208
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>    Affects Versions: 1.0.0
>            Reporter: Randall Hauch
>            Priority: Major
>
> Kafka Connect startup times are excessive with a handful of connectors on the plugin
path or classpath. We should not be scanning three times (once for connectors, once for SMTs,
and once for converters), and hopefully we can avoid scanning directories that are clearly
not plugin directories. 
> We should also consider using Java's Service Loader to quickly identify connectors. The
latter would require a KIP and would require time to for connectors to migrate, but we could
be smarter about only scanning plugin directories that need to be scanned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message