flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Henke <granthe...@gmail.com>
Subject Review Request 51112: FLUME-2972: Handle offset migration in the new Kafka Channel
Date Mon, 15 Aug 2016 21:56:51 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51112/
-----------------------------------------------------------

Review request for Flume.


Bugs: FLUME-2972
    https://issues.apache.org/jira/browse/FLUME-2972


Repository: flume-git


Description
-------

Offsets tracking the position in Kafka consumers change from using zookeeper for offset storage
to Kafka when moving from 0.8.x to 0.9.x.
FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing offsets
get migrated in order to continue consuming where it left off.

This patch adds automated logic on startup to check if Kafka offsets exist, if not and migration
is enabled (by default) then the offsets from Zookeeper are copied and committed to Kafka.


Diffs
-----

  flume-ng-channels/flume-kafka-channel/pom.xml 587b4b4 
  flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
90e3288 
  flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java
ccf46d9 
  flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
b63ac9b 
  flume-ng-doc/sphinx/FlumeUserGuide.rst fde9ff7 

Diff: https://reviews.apache.org/r/51112/diff/


Testing
-------

Unit tests so far.


Thanks,

Grant Henke


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message