kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Giulio Vito de Musso (JIRA)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-5983) Cannot mirror Avro-encoded data using the Apache Kafka MirrorMaker
Date Wed, 27 Sep 2017 13:48:00 GMT
Giulio Vito de Musso created KAFKA-5983:
-------------------------------------------

             Summary: Cannot mirror Avro-encoded data using the Apache Kafka MirrorMaker
                 Key: KAFKA-5983
                 URL: https://issues.apache.org/jira/browse/KAFKA-5983
             Project: Kafka
          Issue Type: Bug
          Components: core
    Affects Versions: 0.11.0.0
         Environment: OS: Linux CentOS 7 and Windows 10
            Reporter: Giulio Vito de Musso


I'm installing an Apache Kafka MirrorMaker instance to replicate one cluster data to one another
cluster. Both on the source and on the target clusters I'm using the Confluent Avro schema
registry and the data is binarized with Avro.

I'm using the latest released version of Confluent 3.3.0 (kafka 0.11). Moreover, the source
broker is on a Windows machine while the target broker is on a Linux machine.
The two Kafka clusters are independent, thus they have different schema registries.

This are my configuration files for the MirrroMaker


{code:title=consumer.properties|borderStyle=solid}
group.id=test-mirrormaker-group
bootstrap.servers=host01:9092
exclude.internal.topics=true
client.id=mirror_maker_consumer0
auto.commit.enabled=false


# Avro schema registry properties
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://host01:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://host01:8081

internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

{code}

{code:title=producer.properties|borderStyle=solid}
bootstrap.servers=host02:9093
compression.type=none
acks=1
client.id=mirror_maker_producer0


# Avro schema registry properties
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://host02:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://host02:8081


internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

{code}

I run the MirrorMaker on the host01 Windows machine with this command

{code}
C:\kafka>.\bin\windows\kafka-mirror-maker.bat --consumer.config .\etc\kafka\consumer.properties
--producer.config .\etc\kafka\producer.properties --whitelist=MY_TOPIC
[2017-09-26 10:09:58,555] WARN The configuration 'internal.key.converter.schemas.enable' was
supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,555] WARN The configuration 'value.converter.schema.registry.url' was
supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,571] WARN The configuration 'internal.key.converter' was supplied but
isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,586] WARN The configuration 'internal.value.converter.schemas.enable'
was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,602] WARN The configuration 'internal.value.converter' was supplied but
isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,633] WARN The configuration 'value.converter' was supplied but isn't
a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,649] WARN The configuration 'key.converter' was supplied but isn't a
known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,649] WARN The configuration 'key.converter.schema.registry.url' was supplied
but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
[2017-09-26 10:09:58,727] WARN The configuration 'internal.key.converter.schemas.enable' was
supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,727] WARN The configuration 'value.converter.schema.registry.url' was
supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,727] WARN The configuration 'internal.key.converter' was supplied but
isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,742] WARN The configuration 'auto.commit.enabled' was supplied but isn't
a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,774] WARN The configuration 'internal.value.converter.schemas.enable'
was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,789] WARN The configuration 'internal.value.converter' was supplied but
isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,805] WARN The configuration 'value.converter' was supplied but isn't
a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,805] WARN The configuration 'key.converter' was supplied but isn't a
known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2017-09-26 10:09:58,821] WARN The configuration 'key.converter.schema.registry.url' was supplied
but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)

{code}

Using the topic UI utility (https://github.com/Landoop/kafka-topics-ui) I can see that on
the target broker the data is sent, but it is shown binarized and I think this is caused by
the misconfiguration of the schema registry.

It seems that the MirrorMaker serializes both key and value data with the _ByteArraySerializer_,
so it ignores the Avro Schema registry case

https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/MirrorMaker.scala#L237

It would be very useful if the Kafka MirrorMaker would read the key/value serialization class
parameters for producer and consumer, allowing to configue the Avro schema serde.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message