flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Grant Henke (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLUME-2972) Handle offset migration in the new Kafka Channel
Date Tue, 16 Aug 2016 19:32:20 GMT

     [ https://issues.apache.org/jira/browse/FLUME-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Grant Henke updated FLUME-2972:
    Attachment: FLUME-2972-2.patch

> Handle offset migration in the new Kafka Channel
> ------------------------------------------------
>                 Key: FLUME-2972
>                 URL: https://issues.apache.org/jira/browse/FLUME-2972
>             Project: Flume
>          Issue Type: Bug
>          Components: Channel
>    Affects Versions: v1.7.0
>            Reporter: Grant Henke
>            Assignee: Grant Henke
>             Fix For: v1.7.0
>         Attachments: FLUME-2972-1.patch, FLUME-2972-2.patch, FLUME-2972.patch
> Offsets tracking the position in Kafka consumers  change from using zookeeper for offset
storage to Kafka when moving from 0.8.x to 0.9.x. 
> FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing
offsets get migrated in order to continue consuming where it left off. 
> Flume should have some automated logic on startup to check if Kafka offsets exist, if
not and migration is enabled (by default) then copy the offsets from Zookeeper and commit
them to Kafka. 

This message was sent by Atlassian JIRA

View raw message