kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-763) Add an option to replica from the largest offset during unclean leader election
Date Fri, 22 Feb 2013 17:52:13 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584479#comment-13584479
] 

Guozhang Wang edited comment on KAFKA-763 at 2/22/13 5:51 PM:
--------------------------------------------------------------

Can these cover all the cases (the bars represents offset range) ?

1. Follower can just remove all its data and start fetching from earliest offset from leader.

new leader    .................|------------------|

follower         ....|------|


2. Follower truncate its log head according to leader's earliest offset (is this necessary?),
and start fetching from the point that is closest to its latest offset.

new leader    .................|------------------|

follower         ................|------|


3. Same to 2.

new leader    .................|------------------|

follower         .....................|------|


4. Follower truncate its log tail, and start fetching from the follower's latest offset.

new leader    .................|------------------|

follower         ................................|------|


5. Follower remove all its data, and start fetching from the follower's earliest offset.

new leader    .................|------------------|

follower         ..........................................|------|


In order to support all these cases, the follower needs to know 1) earliest and latest offset
of the leader, 2) start offset of each segment in order to find the closest offset to its
earliest/latest offset.
                
      was (Author: guozhang):
    Can these cover all the cases (the bars represents offset range) ?

1. Follower can just remove all its data and start fetching from earliest offset from leader.

new leader    .................|------------------|

follower         ....|------|


2. Follower truncate its log head according to leader's earliest offset (is this necessary?),
and start fetching from the point that is closest to its latest offset.

new leader    .................|------------------|

follower         .............|------|


3. Same to 2.

new leader    .................|------------------|

follower         .....................|------|


4. Follower truncate its log tail, and start fetching from the follower's latest offset.

new leader    .................|------------------|

follower         ................................|------|


5. Follower remove all its data, and start fetching from the follower's earliest offset.

new leader    .................|------------------|

follower         ..........................................|------|


In order to support all these cases, the follower needs to know 1) earliest and latest offset
of the leader, 2) start offset of each segment in order to find the closest offset to its
earliest/latest offset.
                  
> Add an option to replica from the largest offset during unclean leader election
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-763
>                 URL: https://issues.apache.org/jira/browse/KAFKA-763
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>    Affects Versions: 0.8
>            Reporter: Jun Rao
>            Assignee: Jun Rao
>            Priority: Blocker
>              Labels: p2
>         Attachments: kafka-763_v1.patch
>
>
> If there is an unclean leader election, a follower may have an offset out of the range
of the leader. Currently, the follower will delete all its data and refetch from the smallest
offset of the leader. It would be useful to add an option to let the follower refetch from
the largest offset of the leader since refetching from the smallest offset may take some time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message