kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6503) Connect: Plugin scan is very slow
Date Tue, 13 Feb 2018 00:03:01 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361616#comment-16361616
] 

ASF GitHub Bot commented on KAFKA-6503:
---------------------------------------

rayokota opened a new pull request #4561: KAFKA-6503: Parallelize plugin scanning
URL: https://github.com/apache/kafka/pull/4561
 
 
   This is a one-liner to parallelize plugin scanning.  This may help in some environments
where otherwise plugin scanning is slow.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Connect: Plugin scan is very slow
> ---------------------------------
>
>                 Key: KAFKA-6503
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6503
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 1.0.0
>            Reporter: Per Steffensen
>            Priority: Critical
>             Fix For: 1.1.0
>
>
> Just upgraded to 1.0.0. It seems some plugin scan has been introduced. It is very slow
- see logs from starting my Kafka-Connect instance at the bottom. It takes almost 4 minutes
scanning. I am running Kafka-Connect in docker based on confluentinc/cp-kafka-connect:4.0.0.
I set plugin.path to /usr/share/java. The only thing I have added is a 13MB jar in /usr/share/java/kafka-connect-file-streamer-client
containing two connectors and a converter. That one alone seems to take 20 secs.
> If it was just scanning in the background, and everything was working it probably would
not be a big issue. But it does not. Right after starting the Kafka-Connect instance I try
to create a connector via the /connectors endpoint, but it will not succeed before the plugin
scanning has finished (4 minutes)
> I am not even sure why scanning is necessary. Is it not always true that connectors,
converters etc are mentioned by name, so to see if it exists, just try to load the class -
the classloader will tell if it is available. Hmmm, there is probably a reason.
> Anyway, either it should be made much faster, or at least Kafka-Connect should be fully
functional (or as functional as possible) while scanning is going on.
> {code}
> [2018-01-30 13:52:26,834] INFO Scanning for plugin classes. This might take a moment
... (org.apache.kafka.connect.cli.ConnectDistributed)
> [2018-01-30 13:52:27,218] INFO Loading plugin from: /usr/share/java/kafka-connect-file-streamer-client
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,037] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-file-streamer-client/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,038] INFO Added plugin 'com.tlt.common.files.streamer.client.kafka.connect.OneTaskPerStreamSourceConnectorManager'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,039] INFO Added plugin 'com.tlt.common.files.streamer.client.kafka.connect.OneTaskPerFilesStreamerServerSourceConnectorManager'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,040] INFO Added plugin 'com.tlt.common.files.streamer.client.kafka.connect.KafkaConnectByteArrayConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:43,049] INFO Loading plugin from: /usr/share/java/kafka-connect-elasticsearch
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:47,595] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-elasticsearch/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:47,611] INFO Added plugin 'io.confluent.connect.elasticsearch.ElasticsearchSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:47,651] INFO Loading plugin from: /usr/share/java/kafka-connect-jdbc
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:49,491] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-jdbc/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:49,491] INFO Added plugin 'io.confluent.connect.jdbc.JdbcSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:49,492] INFO Added plugin 'io.confluent.connect.jdbc.JdbcSourceConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:52:49,663] INFO Loading plugin from: /usr/share/java/kafka-connect-s3
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:51,055] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-s3/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:51,055] INFO Added plugin 'io.confluent.connect.s3.S3SinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:51,061] INFO Added plugin 'io.confluent.connect.storage.tools.SchemaSourceConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:51,064] INFO Added plugin 'io.confluent.connect.avro.AvroConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:53:52,745] INFO Loading plugin from: /usr/share/java/kafka-connect-storage-common
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:25,800] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-storage-common/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:26,402] INFO Loading plugin from: /usr/share/java/kafka-connect-hdfs
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:58,104] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-connect-hdfs/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:58,104] INFO Added plugin 'io.confluent.connect.hdfs.tools.SchemaSourceConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:58,104] INFO Added plugin 'io.confluent.connect.hdfs.HdfsSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:54:58,508] INFO Loading plugin from: /usr/share/java/monitoring-interceptors
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:00,714] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/monitoring-interceptors/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:00,715] INFO Loading plugin from: /usr/share/java/kafka-serde-tools
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:03,900] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/kafka-serde-tools/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:03,911] INFO Loading plugin from: /usr/share/java/confluent-rebalancer
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:11,702] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/confluent-rebalancer/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:11,722] INFO Loading plugin from: /usr/share/java/confluent-control-center
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,874] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/confluent-control-center/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,875] INFO Added plugin 'org.apache.kafka.connect.tools.MockSourceConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,875] INFO Added plugin 'org.apache.kafka.connect.tools.SchemaSourceConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,876] INFO Added plugin 'org.apache.kafka.connect.tools.MockConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,876] INFO Added plugin 'org.apache.kafka.connect.tools.MockSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,876] INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,877] INFO Added plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,878] INFO Added plugin 'org.apache.kafka.connect.converters.ByteArrayConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,878] INFO Added plugin 'org.apache.kafka.connect.storage.StringConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,879] INFO Added plugin 'org.apache.kafka.connect.json.JsonConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,879] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,879] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Key'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,880] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Value'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,880] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Key'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,881] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Value'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,881] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampRouter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,882] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Value'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,882] INFO Added plugin 'org.apache.kafka.connect.transforms.TimestampConverter$Key'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,896] INFO Added plugin 'org.apache.kafka.connect.transforms.MaskField$Key'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,904] INFO Added plugin 'org.apache.kafka.connect.transforms.ExtractField$Key'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,904] INFO Added plugin 'org.apache.kafka.connect.transforms.Flatten$Value'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,905] INFO Added plugin 'org.apache.kafka.connect.transforms.ValueToKey'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,905] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Value'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,905] INFO Added plugin 'org.apache.kafka.connect.transforms.HoistField$Key'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,906] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Value'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,914] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Key'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,914] INFO Added plugin 'org.apache.kafka.connect.transforms.InsertField$Value'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,915] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Value'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,915] INFO Added plugin 'org.apache.kafka.connect.transforms.RegexRouter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,915] INFO Added plugin 'org.apache.kafka.connect.transforms.SetSchemaMetadata$Key'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,915] INFO Added plugin 'org.apache.kafka.connect.transforms.ReplaceField$Key'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:21,919] INFO Loading plugin from: /usr/share/java/confluent-common
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:23,165] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/confluent-common/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:23,175] INFO Loading plugin from: /usr/share/java/kafka (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:41,996] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/kafka/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:41,997] INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:41,998] INFO Added plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:42,021] INFO Loading plugin from: /usr/share/java/confluent-serializers
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:44,081] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/confluent-serializers/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:44,089] INFO Loading plugin from: /usr/share/java/schema-registry (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:49,504] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/schema-registry/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:49,541] INFO Loading plugin from: /usr/share/java/rest-utils (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:55:52,094] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/share/java/rest-utils/}
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,632] INFO Registered loader: sun.misc.Launcher$AppClassLoader@18b4aac2
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,659] INFO Added aliases 'OneTaskPerFilesStreamerServerSourceConnectorManager'
and 'OneTaskPerFilesStreamerServerSource' to plugin 'com.tlt.common.files.streamer.client.kafka.connect.OneTaskPerFilesStreamerServerSourceConnectorManager'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,663] INFO Added aliases 'OneTaskPerStreamSourceConnectorManager'
and 'OneTaskPerStreamSource' to plugin 'com.tlt.common.files.streamer.client.kafka.connect.OneTaskPerStreamSourceConnectorManager'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,665] INFO Added aliases 'ElasticsearchSinkConnector' and 'ElasticsearchSink'
to plugin 'io.confluent.connect.elasticsearch.ElasticsearchSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,667] INFO Added aliases 'HdfsSinkConnector' and 'HdfsSink' to plugin
'io.confluent.connect.hdfs.HdfsSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,668] INFO Added aliases 'JdbcSinkConnector' and 'JdbcSink' to plugin
'io.confluent.connect.jdbc.JdbcSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,670] INFO Added aliases 'JdbcSourceConnector' and 'JdbcSource' to
plugin 'io.confluent.connect.jdbc.JdbcSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,672] INFO Added aliases 'S3SinkConnector' and 'S3Sink' to plugin
'io.confluent.connect.s3.S3SinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,673] INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink'
to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,695] INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource'
to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,697] INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,699] INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin
'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,700] INFO Added aliases 'MockSourceConnector' and 'MockSource' to
plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,702] INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink'
to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,705] INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource'
to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,706] INFO Added aliases 'KafkaConnectByteArrayConverter' and 'KafkaConnectByteArray'
to plugin 'com.tlt.common.files.streamer.client.kafka.connect.KafkaConnectByteArrayConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,711] INFO Added aliases 'AvroConverter' and 'Avro' to plugin 'io.confluent.connect.avro.AvroConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,711] INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to
plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,715] INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,715] INFO Added aliases 'StringConverter' and 'String' to plugin
'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,727] INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,738] INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> [2018-01-30 13:56:14,739] INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey'
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message