kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Ranson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-3772) MirrorMaker crashes on Corrupted Message
Date Tue, 31 May 2016 15:30:12 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

James Ranson updated KAFKA-3772:
--------------------------------
    Description: 
We recently came across an issue where a message on our source kafka cluster became corrupted.
When MirrorMaker tried to consume this message, the thread crashed and caused the entire process
to also crash. Each time we attempted to restart MM, it crashed on the same message. There
is no information in the MM logs about which message it was trying to consume (what topic,
what offset, etc). So the only way we were able to get past the issue was to go into the zookeeper
tree for our mirror consumer group and increment the offset for every partition on every topic
until the MM process could start without crashing. This is not a tenable operational solution.
MirrorMaker should gracefully skip corrupt messages since they will never be able to be replicated
anyway.

{noformat}2016-05-26 20:02:26,787 FATAL  MirrorMaker$MirrorMakerThread - [{}] [mirrormaker-thread-3]
Mirror maker thread failure due to
kafka.message.InvalidMessageException: Message is corrupt (stored crc = 33747148, computed
crc = 3550736267)
	at kafka.message.Message.ensureValid(Message.scala:167)
	at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:101)
	at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
	at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
	at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
	at kafka.tools.MirrorMaker$MirrorMakerOldConsumer.hasData(MirrorMaker.scala:483)
	at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:394)

2016-05-26 20:02:27,580 FATAL  MirrorMaker$MirrorMakerThread - [{}] [mirrormaker-thread-3]
Mirror maker thread exited abnormally, stopping the whole mirror maker.{noformat}

  was:
We recently came across an issue where a message on our source kafka cluster became corrupted.
When MirrorMaker tried to consume this message, the thread crashed and caused the entire process
to also crash. Each time we attempted to restart MM, it crashed on the same message. There
is no information in the MM logs about which message it was trying to consume (what topic,
what offset, etc). So the only way we were able to get past the issue was to go into the zookeeper
tree for our mirror consumer group and increment the offset for every partition on every topic
until the MM process could start without crashing. This is not a tenable operational solution.
MirrorMaker should gracefully skip corrupt messages since they will never be able to be replicated
anyway.

```2016-05-26 20:02:26,787 FATAL  MirrorMaker$MirrorMakerThread - [{}] [mirrormaker-thread-3]
Mirror maker thread failure due to
kafka.message.InvalidMessageException: Message is corrupt (stored crc = 33747148, computed
crc = 3550736267)
	at kafka.message.Message.ensureValid(Message.scala:167)
	at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:101)
	at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
	at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
	at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
	at kafka.tools.MirrorMaker$MirrorMakerOldConsumer.hasData(MirrorMaker.scala:483)
	at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:394)

2016-05-26 20:02:27,580 FATAL  MirrorMaker$MirrorMakerThread - [{}] [mirrormaker-thread-3]
Mirror maker thread exited abnormally, stopping the whole mirror maker.```


> MirrorMaker crashes on Corrupted Message
> ----------------------------------------
>
>                 Key: KAFKA-3772
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3772
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.9.0.1
>            Reporter: James Ranson
>              Labels: mirror-maker
>
> We recently came across an issue where a message on our source kafka cluster became corrupted.
When MirrorMaker tried to consume this message, the thread crashed and caused the entire process
to also crash. Each time we attempted to restart MM, it crashed on the same message. There
is no information in the MM logs about which message it was trying to consume (what topic,
what offset, etc). So the only way we were able to get past the issue was to go into the zookeeper
tree for our mirror consumer group and increment the offset for every partition on every topic
until the MM process could start without crashing. This is not a tenable operational solution.
MirrorMaker should gracefully skip corrupt messages since they will never be able to be replicated
anyway.
> {noformat}2016-05-26 20:02:26,787 FATAL  MirrorMaker$MirrorMakerThread - [{}] [mirrormaker-thread-3]
Mirror maker thread failure due to
> kafka.message.InvalidMessageException: Message is corrupt (stored crc = 33747148, computed
crc = 3550736267)
> 	at kafka.message.Message.ensureValid(Message.scala:167)
> 	at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:101)
> 	at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
> 	at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
> 	at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
> 	at kafka.tools.MirrorMaker$MirrorMakerOldConsumer.hasData(MirrorMaker.scala:483)
> 	at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:394)
> 2016-05-26 20:02:27,580 FATAL  MirrorMaker$MirrorMakerThread - [{}] [mirrormaker-thread-3]
Mirror maker thread exited abnormally, stopping the whole mirror maker.{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message