Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Date: Wed, 12 Apr 2017 16:50:41 +0000 (UTC)
From: "Ismael Juma (JIRA)" <jira@apache.org>
To: dev@kafka.apache.org
Message-ID: <JIRA.13063544.1492014935000.267232.1492015841627@Atlassian.JIRA>
In-Reply-To: <JIRA.13063544.1492014935000@Atlassian.JIRA>
References: <JIRA.13063544.1492014935000@Atlassian.JIRA> <JIRA.13063544.1492014935596@jira-lw-us.apache.org>
Subject: [jira] [Updated] (KAFKA-5060) Offset not found while broker is
 rebuilding its index after an index corruption
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 12 Apr 2017 16:50:51 -0000


     [ https://issues.apache.org/jira/browse/KAFKA-5060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ismael Juma updated KAFKA-5060:
-------------------------------
    Labels: reliability  (was: )

> Offset not found while broker is rebuilding its index after an index corruption
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-5060
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5060
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.10.1.0
>            Reporter: Romaric Parmentier
>            Priority: Critical
>              Labels: reliability
>
> After rebooting our kafka servers to change a configuration, one of my consumers running old consumer has fail to find a new leader for a period of 15 minutes. The topic has a replication factor of 2.
> When the spare server has finally been found and elected leader, the previous consumed offset was not able to be found because the broker was rebuilding index. 
> So my consumer has decided to follow the configuration auto.offset.reset which is pretty bad because the offset will exist 2 minutes later:
> 2017-04-12 14:59:08,568] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (/var/lib/kafka/my_topic-6/00000000130248110337.index) has non-zero size but the last offset is 130248110337 which is no larger than the base offset 130248110337.}. deleting /var/lib/kafka/my_topic-6/00000000130248110337.timeindex, /var/lib/kafka/my_topic-6/00000000130248110337.index and rebuilding index... (kafka.log.Log)
> [2017-04-12 15:01:41,490] INFO Completed load of log my_topic-6 with 6146 log segments and log end offset 130251895436 in 169696 ms (kafka.log.Log)
> Maybe it is handled by the new consumer or there is a some configuration to handle this case but I didn't find anything


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)