Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
Date: Thu, 7 Jul 2016 17:11:11 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: dev@apex.incubator.apache.org
Message-ID: <JIRA.12968045.1463002767000.49820.1467911471167@Atlassian.JIRA>
In-Reply-To: <JIRA.12968045.1463002767000@Atlassian.JIRA>
References: <JIRA.12968045.1463002767000@Atlassian.JIRA> <JIRA.12968045.1463002767176@arcas>
Subject: [jira] [Commented] (APEXMALHAR-2086) Kafka Output Operator with
 Kafka 0.9 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 07 Jul 2016 17:11:17 -0000


    [ https://issues.apache.org/jira/browse/APEXMALHAR-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366420#comment-15366420 ] 

ASF GitHub Bot commented on APEXMALHAR-2086:
--------------------------------------------

GitHub user sandeshh reopened a pull request:

    https://github.com/apache/apex-malhar/pull/298

    [APEXMALHAR-2086] Kafka output operator: 0.9.0

    Kafka output exactly once operator and the regular output operator.
    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sandeshh/apex-malhar APEXMALHAR-2086

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/apex-malhar/pull/298.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #298
    
----
commit 6bf316da76bf1d1871f52a114a307c05ff652376
Author: sandeshh <sandesh.hegde@gmail.com>
Date:   2016-05-25T15:56:56Z

    Kafka 0.9.0 output operators and unit tests.
    
    1. Abstract Base class
    2. Kafka Output operator
    3. Exactly Once output operator
         Key in the Kafka message is used by the operator to track the tuples written by it.

----


> Kafka Output Operator with Kafka 0.9 API
> ----------------------------------------
>
>                 Key: APEXMALHAR-2086
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2086
>             Project: Apache Apex Malhar
>          Issue Type: New Feature
>            Reporter: Sandesh
>            Assignee: Sandesh
>
> Goal : 2 Operartors for Kafka Output
>       1. Simple Kafka Output Operator 
>             - Supports Atleast Once 
>             - Expose most used producer properties as class properties
>       2. Exactly Once Kafka Output ( Not possible in all the cases, will be documented later )
>             
> Design for Exactly Once
> Window Data Manager - Stores the Kafka partitions offsets.
> Kafka Key - Used by the operator = AppID#OperatorId
> During recovery. Partially written window is re-created using the following  approach:
> Tuples between the largest recovery offsets and the current offset are checked. Based on the key, tuples written by the other entities are discarded. 
> Only tuples which are not in the recovered set are emitted.
> Tuples needs to be unique within the window.
>       


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)