Return-Path: X-Original-To: apmail-kafka-dev-archive@www.apache.org Delivered-To: apmail-kafka-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E2CA89797 for ; Fri, 22 Feb 2013 17:54:12 +0000 (UTC) Received: (qmail 19359 invoked by uid 500); 22 Feb 2013 17:54:12 -0000 Delivered-To: apmail-kafka-dev-archive@kafka.apache.org Received: (qmail 19298 invoked by uid 500); 22 Feb 2013 17:54:12 -0000 Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kafka.apache.org Delivered-To: mailing list dev@kafka.apache.org Received: (qmail 19286 invoked by uid 99); 22 Feb 2013 17:54:12 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Feb 2013 17:54:12 +0000 Date: Fri, 22 Feb 2013 17:54:12 +0000 (UTC) From: "Guozhang Wang (JIRA)" To: dev@kafka.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (KAFKA-763) Add an option to replica from the largest offset during unclean leader election MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/KAFKA-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584479#comment-13584479 ] Guozhang Wang edited comment on KAFKA-763 at 2/22/13 5:52 PM: -------------------------------------------------------------- Can these cover all the cases (the bars represents offset range) ? 1. Follower can just remove all its data and start fetching from earliest offset from leader. new leader .................|------------------| follower ....|------| 2. Follower truncate its log head according to leader's earliest offset (is this necessary?), and start fetching from the point that is closest to its latest offset. new leader .................|------------------| follower ..................|------| 3. Same to 2. new leader .................|------------------| follower ..........................|------| 4. Follower truncate its log tail, and start fetching from the follower's latest offset. new leader .................|------------------| follower .....................................|------| 5. Follower remove all its data, and start fetching from the follower's earliest offset. new leader .................|------------------| follower ...............................................|------| In order to support all these cases, the follower needs to know 1) earliest and latest offset of the leader, 2) start offset of each segment in order to find the closest offset to its earliest/latest offset. was (Author: guozhang): Can these cover all the cases (the bars represents offset range) ? 1. Follower can just remove all its data and start fetching from earliest offset from leader. new leader .................|------------------| follower ....|------| 2. Follower truncate its log head according to leader's earliest offset (is this necessary?), and start fetching from the point that is closest to its latest offset. new leader .................|------------------| follower ................|------| 3. Same to 2. new leader .................|------------------| follower .....................|------| 4. Follower truncate its log tail, and start fetching from the follower's latest offset. new leader .................|------------------| follower ................................|------| 5. Follower remove all its data, and start fetching from the follower's earliest offset. new leader .................|------------------| follower ..........................................|------| In order to support all these cases, the follower needs to know 1) earliest and latest offset of the leader, 2) start offset of each segment in order to find the closest offset to its earliest/latest offset. > Add an option to replica from the largest offset during unclean leader election > ------------------------------------------------------------------------------- > > Key: KAFKA-763 > URL: https://issues.apache.org/jira/browse/KAFKA-763 > Project: Kafka > Issue Type: Improvement > Components: core > Affects Versions: 0.8 > Reporter: Jun Rao > Assignee: Jun Rao > Priority: Blocker > Labels: p2 > Attachments: kafka-763_v1.patch > > > If there is an unclean leader election, a follower may have an offset out of the range of the leader. Currently, the follower will delete all its data and refetch from the smallest offset of the leader. It would be useful to add an option to let the follower refetch from the largest offset of the leader since refetching from the smallest offset may take some time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira