kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rémi REY (JIRA) <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6177) kafka-mirror-maker.sh RecordTooLargeException
Date Mon, 13 Nov 2017 14:12:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249639#comment-16249639
] 

Rémi REY commented on KAFKA-6177:
---------------------------------

Hello all,

Anyone to acknowledge this ticket please ?

Thank you.

> kafka-mirror-maker.sh RecordTooLargeException
> ---------------------------------------------
>
>                 Key: KAFKA-6177
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6177
>             Project: Kafka
>          Issue Type: Bug
>          Components: producer 
>    Affects Versions: 0.10.1.1
>         Environment: centos 7
>            Reporter: Rémi REY
>            Priority: Minor
>              Labels: support
>         Attachments: consumer.config, producer.config, server.properties
>
>
> Hi all,
> I am facing an issue with kafka-mirror-maker.sh.
> We have 2 kafka clusters with the same configuration and mirror maker instances in charge
of the mirroring between the clusters.
> We haven't change the default configuration on the message size, so the 1000012 bytes
limitation is expected on both clusters.
> we are facing the following error at the mirroring side:
> {code}
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: [2017-09-21 14:30:49,431] ERROR Error
when sending message to topic my_topic_name with key: 81 bytes, value: 1000272 bytes with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: org.apache.kafka.common.errors.RecordTooLargeException:
The request included a message larger than the max message size the server will accept.
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: [2017-09-21 14:30:49,511] ERROR Error
when sending message to topic my_topic_name with key: 81 bytes, value: 13846 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: java.lang.IllegalStateException: Producer
is closed forcefully.
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:513)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:493)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:156)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at java.lang.Thread.run(Thread.java:745)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: [2017-09-21 14:30:49,511] FATAL [mirrormaker-thread-0]
Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMakerThread)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: java.lang.IllegalStateException: Cannot
send after the producer is closed.
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:185)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:474)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:436)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:657)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$6.apply(MirrorMaker.scala:434)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$6.apply(MirrorMaker.scala:434)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:434)
> {code}
> Why am I getting this error ? 
> {code}
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: [2017-09-21 14:30:49,431] ERROR Error
when sending message to topic my_topic_name with key: 81 bytes, value: 1000272 bytes with
error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> Sep 21 14:30:49 lpa2e194 kafka-mirror-maker.sh: org.apache.kafka.common.errors.RecordTooLargeException:
The request included a message larger than the max message size the server will accept.
> {code}
> How can mirror maker encounter a 1000272 bytes message while the kafka cluster being
mirrored has the default limitation of 1000012 bytes for a message ?
> Find the mirrormaker consumer and producer config files attached.
> Thanks for your inputs.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message