flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-2983) Handle offset migration in the new Kafka Source
Date Thu, 01 Sep 2016 02:59:20 GMT

    [ https://issues.apache.org/jira/browse/FLUME-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454127#comment-15454127

Hudson commented on FLUME-2983:

FAILURE: Integrated in Jenkins build Flume-trunk-hbase-1 #202 (See [https://builds.apache.org/job/Flume-trunk-hbase-1/202/])
FLUME-2983. Handle offset migration in the new Kafka Source (mpercy: [http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=330e572879c9903f5b1084d84b237685aa9970c6])
* (edit) flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
* (edit) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSource.java
* (edit) flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
* (edit) flume-ng-sources/flume-kafka-source/pom.xml
* (edit) flume-ng-doc/sphinx/FlumeUserGuide.rst
* (edit) flume-ng-sources/flume-kafka-source/src/main/java/org/apache/flume/source/kafka/KafkaSourceConstants.java
* (edit) flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/TestKafkaSource.java
* (edit) flume-ng-sources/flume-kafka-source/src/test/java/org/apache/flume/source/kafka/KafkaSourceEmbeddedKafka.java

> Handle offset migration in the new Kafka Source
> -----------------------------------------------
>                 Key: FLUME-2983
>                 URL: https://issues.apache.org/jira/browse/FLUME-2983
>             Project: Flume
>          Issue Type: Bug
>    Affects Versions: v1.7.0
>            Reporter: Grant Henke
>            Assignee: Grant Henke
>             Fix For: v1.7.0
> Similar to FLUME-2972, Offsets tracking the position in Kafka consumers change from using
zookeeper for offset storage to Kafka when moving from 0.8.x to 0.9.x.
> FLUME-2821 makes the client change in the Kafka Source but does not ensure existing offsets
get migrated in order to continue consuming where it left off.
> Flume should have some automated logic on startup to check if Kafka offsets exist, if
not and migration is enabled (by default) then copy the offsets from Zookeeper and commit
them to Kafka.
> This change should also fix the backwards incompatibility caused by removing the zookeeperConnect
property. The bootstrap can be looked up if zookeeperConnect is used. 

This message was sent by Atlassian JIRA

View raw message