kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julius ┼Żaromskis (JIRA) <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-4972) Kafka 0.10.0 Found a corrupted index file during Kafka broker startup
Date Mon, 18 Sep 2017 07:23:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16169681#comment-16169681
] 

Julius ┼Żaromskis edited comment on KAFKA-4972 at 9/18/17 7:22 AM:
------------------------------------------------------------------

There's a bunch of warning msgs in my log file, kafka is slow to restart. Upgrading 0.10.2
to 0.11.0.1

{{[2017-09-18 06:53:19,349] WARN Found a corrupted index file due to requirement failed: Corrupt
index found, index file (/var/kafka/dispatch.task-ack-6/00000000000000021796.index) has non-zero
size but the last offset is 21796 which is no larger than the base offset 21796.}. deleting
/var/kafka/dispatch.task-ack-6/00000000000000021796.timeindex, /var/kafka/dispatch.task-ack-6/00000000000000021796.index,
and /var/kafka/dispatch.task-ack-6/00000000000000021796.txnindex and rebuilding index... (kafka.log.Log)
[2017-09-18 06:56:10,533] WARN Found a corrupted index file due to requirement failed: Corrupt
index found, index file (/var/kafka/dispatch.task-ack-10/00000000000000027244.index) has non-zero
size but the last offset is 27244 which is no larger than the base offset 27244.}. deleting
/var/kafka/dispatch.task-ack-10/00000000000000027244.timeindex, /var/kafka/dispatch.task-ack-10/00000000000000027244.index,
and /var/kafka/dispatch.task-ack-10/00000000000000027244.txnindex and rebuilding index...
(kafka.log.Log)
[2017-09-18 07:09:17,710] WARN Found a corrupted index file due to requirement failed: Corrupt
index found, index file (/var/kafka/dispatch.status-3/00000000000049362755.index) has non-zero
size but the last offset is 49362755 which is no larger than the base offset 49362755.}. deleting
/var/kafka/dispatch.status-3/00000000000049362755.timeindex, /var/kafka/dispatch.status-3/00000000000049362755.index,
and /var/kafka/dispatch.status-3/00000000000049362755.txnindex and rebuilding index... (kafka.log.Log)}}


was (Author: juliuszaromskis):
There's a bunch of warning msgs in my log file, kafka is slow to restart

{{[2017-09-18 06:53:19,349] WARN Found a corrupted index file due to requirement failed: Corrupt
index found, index file (/var/kafka/dispatch.task-ack-6/00000000000000021796.index) has non-zero
size but the last offset is 21796 which is no larger than the base offset 21796.}. deleting
/var/kafka/dispatch.task-ack-6/00000000000000021796.timeindex, /var/kafka/dispatch.task-ack-6/00000000000000021796.index,
and /var/kafka/dispatch.task-ack-6/00000000000000021796.txnindex and rebuilding index... (kafka.log.Log)
[2017-09-18 06:56:10,533] WARN Found a corrupted index file due to requirement failed: Corrupt
index found, index file (/var/kafka/dispatch.task-ack-10/00000000000000027244.index) has non-zero
size but the last offset is 27244 which is no larger than the base offset 27244.}. deleting
/var/kafka/dispatch.task-ack-10/00000000000000027244.timeindex, /var/kafka/dispatch.task-ack-10/00000000000000027244.index,
and /var/kafka/dispatch.task-ack-10/00000000000000027244.txnindex and rebuilding index...
(kafka.log.Log)
[2017-09-18 07:09:17,710] WARN Found a corrupted index file due to requirement failed: Corrupt
index found, index file (/var/kafka/dispatch.status-3/00000000000049362755.index) has non-zero
size but the last offset is 49362755 which is no larger than the base offset 49362755.}. deleting
/var/kafka/dispatch.status-3/00000000000049362755.timeindex, /var/kafka/dispatch.status-3/00000000000049362755.index,
and /var/kafka/dispatch.status-3/00000000000049362755.txnindex and rebuilding index... (kafka.log.Log)}}

> Kafka 0.10.0  Found a corrupted index file during Kafka broker startup
> ----------------------------------------------------------------------
>
>                 Key: KAFKA-4972
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4972
>             Project: Kafka
>          Issue Type: Bug
>          Components: log
>    Affects Versions: 0.10.0.0
>         Environment: JDK: HotSpot  x64  1.7.0_80
> Tag: 0.10.0
>            Reporter: fangjinuo
>            Priority: Critical
>             Fix For: 0.11.0.2
>
>         Attachments: Snap3.png
>
>
> After force shutdown all kafka brokers one by one, restart them one by one, but a broker
startup failure.
> The following WARN leval log was found in the log file:
> found a corrutped index file,  xxxx.index , delet it  ...
> you can view details by following attachment.
> I look up some codes in core module, found out :
> the nonthreadsafe method LogSegment.append(offset, messages)  has tow caller:
> 1) Log.append(messages)                          // here has a synchronized lock 
> 2) LogCleaner.cleanInto(topicAndPartition, source, dest, map, retainDeletes, messageFormatVersion)
  // here has not 
> So I guess this may be the reason for the repeated offset in 00000xx.log file (logsegment's
.log) 
> Although this is just my inference, but I hope that this problem can be quickly repaired



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message