flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Percy <mpe...@apache.org>
Subject Re: Review Request 52598: FLUME-2999 - Kafka channel and sink should enable statically assigned partition per event via header
Date Mon, 10 Oct 2016 18:12:23 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52598/#review152027
-----------------------------------------------------------


Ship it!




Ship It!

- Mike Percy


On Oct. 10, 2016, 1:53 a.m., Tristan Stevens wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52598/
> -----------------------------------------------------------
> 
> (Updated Oct. 10, 2016, 1:53 a.m.)
> 
> 
> Review request for Flume and Grant Henke.
> 
> 
> Repository: flume-git
> 
> 
> Description
> -------
> 
> This feature is useful for anyone who needs greater control of which partitions are being
written to - normally in a situation where multiple Flume agents are being deployed in order
to horizontally scale, or alternatively if there is a scenario where there is a skew in data
that might lead to one or more partitions hotspotting.
> We also have the ability to specify custom partitions on to the Kafka Producer itself
using the kafka.* configuration properties.
> 
> The Kafka Producer provides the ability to set the partition ID using the following constructor
(https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#ProducerRecord%28java.lang.String,%20java.lang.Integer,%20K,%20V%29
), this is just a matter of providing the option to use this constructor.
> 
> This is specified in one of two ways: either via the staticPartition configuration property,
which means that every message goes to the specified partition, or via the partitionHeader
configuration property, which directs the implementation to retrieve the partitionId from
one of the event headers.
> 
> 
> Diffs
> -----
> 
>   flume-ng-channels/flume-kafka-channel/pom.xml c1cc844 
>   flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannel.java
66b553a 
>   flume-ng-channels/flume-kafka-channel/src/main/java/org/apache/flume/channel/kafka/KafkaChannelConfiguration.java
3ab807b 
>   flume-ng-channels/flume-kafka-channel/src/test/java/org/apache/flume/channel/kafka/TestKafkaChannel.java
57c0b28 
>   flume-ng-doc/sphinx/FlumeUserGuide.rst ab71d38 
>   flume-ng-sinks/flume-ng-kafka-sink/pom.xml 195c921 
>   flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSink.java
89bdd84 
>   flume-ng-sinks/flume-ng-kafka-sink/src/main/java/org/apache/flume/sink/kafka/KafkaSinkConstants.java
1bf380c 
>   flume-ng-sinks/flume-ng-kafka-sink/src/test/java/org/apache/flume/sink/kafka/TestKafkaSink.java
76eca37 
>   flume-ng-sources/flume-kafka-source/pom.xml c89ea1a 
>   flume-shared/flume-shared-kafka-test/pom.xml PRE-CREATION 
>   flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/KafkaPartitionTestUtil.java
PRE-CREATION 
>   flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionOption.java
PRE-CREATION 
>   flume-shared/flume-shared-kafka-test/src/main/java/org/apache/flume/shared/kafka/test/PartitionTestScenario.java
PRE-CREATION 
>   flume-shared/pom.xml PRE-CREATION 
>   pom.xml 2332a29 
> 
> Diff: https://reviews.apache.org/r/52598/diff/
> 
> 
> Testing
> -------
> 
> Unit testing done for both Kafka Channel and Kafka Sink.
> 
> 
> Thanks,
> 
> Tristan Stevens
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message