apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2459) KafkaInputoperator using 0.10.* Kafka consumer API
Date Wed, 05 Apr 2017 10:08:41 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15956619#comment-15956619

ASF GitHub Bot commented on APEXMALHAR-2459:

GitHub user chaithu14 opened a pull request:


    APEXMALHAR-2459 1)Refactor the existing Kafka Input Operator. 2)Added the support of KafkaInputOperator
using 0.10 consumer API


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/chaithu14/incubator-apex-malhar APEXMALHAR-2459-Kafkainput

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #600


> KafkaInputoperator using 0.10.* Kafka consumer API
> --------------------------------------------------
>                 Key: APEXMALHAR-2459
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2459
>             Project: Apache Apex Malhar
>          Issue Type: Task
>            Reporter: Chaitanya
>            Assignee: Chaitanya
> Changes in version of Kafka
> -------------------------------------------------
>    Message format in 0.10.0 has changed. Messages now include a time-stamp field and
compressed messages include relative offsets. 
>    Please refer the below link about the changes in Kafka version:
> https://kafka.apache.org/documentation/#upgrade_10_breaking.
> Design
> ---------
>   I'm proposing the below design for new Kafka connectors by refactoring the existing
Kafka connector(0.9.* version)
> 1) Split up the malhar-kafka to malhar-kafka-common, malhar-kafka_0.9, malhar-kafka_0.10.
Directory structure would be as follows:
>   malhar-kafka
>           |--malhar-kafka-common
>           |--malhar-kafka_0.9
>           |--malhar-kafka_0.10
> If possible, will add the connectors for 0.8.x version.
> 2) Convert the KafkaConsumerWrapper to AbstractKafkaConsumerWrapper by adding the abstract
methods over the consumer API's.
> 3) "malhar-kafka-common" package consists of the abstract classes and by default it builds
over the 0.9.x version of Kafka.
> 4) "malhar-kafka-0.*" consists of concrete classes and its corresponding KafkaConsumerWrapper.

This message was sent by Atlassian JIRA

View raw message