apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chaitanya Chebolu <chaita...@datatorrent.com>
Subject Re: Kafka Input Operator for 0.10.x
Date Wed, 29 Mar 2017 08:59:50 GMT
Created JIRA : https://issues.apache.org/jira/browse/APEXMALHAR-2459


On Wed, Mar 22, 2017 at 4:11 PM, Chaitanya Chebolu <
chaitanya@datatorrent.com> wrote:

> Hi All,
>
>    I would like to propose the Kafka Input Operator using 0.10.0.0
> consumer APIs.
>
> Changes in 0.10.0.0 version of Kafka
> -------------------------------------------------
>    Message format in 0.10.0 has changed. Messages now include a time-stamp
> field and compressed messages include relative offsets.
>
>    Please refer the below link about the changes in Kafka 0.10.0.0 version:
> https://kafka.apache.org/documentation/#upgrade_10_breaking.
>
> Design
> ---------
>   I'm proposing the below design for new Kafka connectors by refactoring
> the existing Kafka connector(0.9.* version)
> 1) Split up the malhar-kafka to malhar-kafka-common, malhar-kafka_0.9,
> malhar-kafka_0.10. Directory structure would be as follows:
>   malhar-kafka
>           |--malhar-kafka-common
>           |--malhar-kafka_0.9
>           |--malhar-kafka_0.10
> If possible, will add the connectors for 0.8.x version.
> 2) Convert the KafkaConsumerWrapper to AbstractKafkaConsumerWrapper by
> adding the abstract methods over the consumer API's.
> 3) "malhar-kafka-common" package consists of the abstract classes and by
> default it builds over the 0.9.x version of Kafka.
> 4) "malhar-kafka-0.*" consists of concrete classes and its corresponding
> KafkaConsumerWrapper.
>
>   Please share your thoughts on design.
>
> Regards,
> Chaitanya
>
> --
>
> *Chaitanya*
>
> Software Engineer
>
> E: chaitanya@datatorrent.com | Twitter: @chaithu1403
>
> www.datatorrent.com  |  apex.apache.org
>
>
>


-- 

*Chaitanya*

Software Engineer

E: chaitanya@datatorrent.com | Twitter: @chaithu1403

www.datatorrent.com  |  apex.apache.org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message