kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Spiros Ioannou (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5060) Offset not found while broker is rebuilding its index after an index corruption
Date Wed, 30 Aug 2017 12:20:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147141#comment-16147141
] 

Spiros Ioannou commented on KAFKA-5060:
---------------------------------------

We get the same error on every restart on kafka 0.11:

{noformat}

[2017-08-30 12:08:12,970] INFO Loading producer state from offset 1012 for partition i2SvarEvts-851
with message format version 2 (kafka.log.Log)
[2017-08-30 12:08:12,970] INFO Loading producer state from snapshot file 00000000000000001012.snapshot
for partition i2SvarEvts-851 (kafka.log.ProducerStateManager)
[2017-08-30 12:08:12,970] INFO Completed load of log i2SvarEvts-851 with 1 log segments, log
start offset 0 and log end offset 1012 in 11 ms (kafka.log.Log)
[2017-08-30 12:08:12,973] WARN Found a corrupted index file due to requirement failed: Corrupt
index found, index file (/disk1/kafka/kafka-logs/i2SvarEvts-5/00000000000000000000.index)
has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting
/disk1/kafka/kafka-logs/i2SvarEvts-5/00000000000000000000.timeindex, /disk1/kafka/kafka-logs/i2SvarEvts-5/00000000000000000000.index,
and /disk1/kafka/kafka-logs/i2SvarEvts-5/00000000000000000000.txnindex and rebuilding index...
(kafka.log.Log)
[2017-08-30 12:08:12,973] INFO Recovering unflushed segment 0 in log i2SvarEvts-5. (kafka.log.Log)
[2017-08-30 12:08:12,973] INFO Loading producer state from offset 0 for partition i2SvarEvts-5
with message format version 2 (kafka.log.Log)
[2017-08-30 12:08:12,974] INFO Completed load of log i2SvarEvts-5 with 1 log segments, log
start offset 0 and log end offset 0 in 3 ms (kafka.log.Log)
[2017-08-30 12:08:12,976] WARN Found a corrupted index file due to requirement failed: Corrupt
index found, index file (/disk1/kafka/kafka-logs/i2SvarEvts-167/00000000000000000000.index)
has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting
/disk1/kafka/kafka-logs/i2SvarEvts-167/00000000000000000000.timeindex, /disk1/kafka/kafka-logs/i2SvarEvts-167/00000000000000000000.i
ndex, and /disk1/kafka/kafka-logs/i2SvarEvts-167/00000000000000000000.txnindex and rebuilding
index... (kafka.log.Log)
[2017-08-30 12:08:12,977] INFO Recovering unflushed segment 0 in log i2SvarEvts-167. (kafka.log.Log)
[2017-08-30 12:08:12,977] INFO Loading producer state from offset 0 for partition i2SvarEvts-167
with message format version 2 (kafka.log.Log)
[2017-08-30 12:08:12,977] INFO Completed load of log i2SvarEvts-167 with 1 log segments, log
start offset 0 and log end offset 0 in 3 ms (kafka.log.Log)
[2017-08-30 12:08:12,980] WARN Found a corrupted index file due to requirement failed: Corrupt
index found, index file (/disk1/kafka/kafka-logs/i2SvarEvts-300/00000000000000000000.index)
has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting
/disk1/kafka/kafka-logs/i2SvarEvts-300/00000000000000000000.timeindex, /disk1/kafka/kafka-logs/i2SvarEvts-300/00000000000000000000.i
ndex, and /disk1/kafka/kafka-logs/i2SvarEvts-300/00000000000000000000.txnindex and rebuilding
index... (kafka.log.Log)
[2017-08-30 12:08:12,999] INFO Recovering unflushed segment 0 in log i2SvarEvts-300. (kafka.log.Log)
[2017-08-30 12:08:13,021] INFO Loading producer state from offset 7167 for partition i2SvarEvts-300
with message format version 2 (kafka.log.Log)
[2017-08-30 12:08:13,021] INFO Loading producer state from snapshot file 00000000000000007167.snapshot
for partition i2SvarEvts-300 (kafka.log.ProducerStateManager)
[2017-08-30 12:08:13,021] INFO Completed load of log i2SvarEvts-300 with 1 log segments, log
start offset 0 and log end offset 7167 in 43 ms (kafka.log.Log)
...

{noformat}


> Offset not found while broker is rebuilding its index after an index corruption
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-5060
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5060
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.10.1.0
>            Reporter: Romaric Parmentier
>            Priority: Critical
>              Labels: reliability
>
> After rebooting our kafka servers to change a configuration, one of my consumers running
old consumer has fail to find a new leader for a period of 15 minutes. The topic has a replication
factor of 2.
> When the spare server has finally been found and elected leader, the previous consumed
offset was not able to be found because the broker was rebuilding index. 
> So my consumer has decided to follow the configuration auto.offset.reset which is pretty
bad because the offset will exist 2 minutes later:
> 2017-04-12 14:59:08,568] WARN Found a corrupted index file due to requirement failed:
Corrupt index found, index file (/var/lib/kafka/my_topic-6/00000000130248110337.index) has
non-zero size but the last offset is 130248110337 which is no larger than the base offset
130248110337.}. deleting /var/lib/kafka/my_topic-6/00000000130248110337.timeindex, /var/lib/kafka/my_topic-6/00000000130248110337.index
and rebuilding index... (kafka.log.Log)
> [2017-04-12 15:01:41,490] INFO Completed load of log my_topic-6 with 6146 log segments
and log end offset 130251895436 in 169696 ms (kafka.log.Log)
> Maybe it is handled by the new consumer or there is a some configuration to handle this
case but I didn't find anything



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message