Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
Date: Fri, 1 Jul 2016 17:07:11 +0000 (UTC)
From: "Sandesh (JIRA)" <jira@apache.org>
To: dev@apex.incubator.apache.org
Message-ID: <JIRA.12968045.1463002767000.6553.1467392831782@Atlassian.JIRA>
In-Reply-To: <JIRA.12968045.1463002767000@Atlassian.JIRA>
References: <JIRA.12968045.1463002767000@Atlassian.JIRA> <JIRA.12968045.1463002767176@arcas>
Subject: [jira] [Commented] (APEXMALHAR-2086) Kafka Output Operator with
 Kafka 0.9 API
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 01 Jul 2016 17:07:15 -0000


    [ https://issues.apache.org/jira/browse/APEXMALHAR-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359287#comment-15359287 ] 

Sandesh commented on APEXMALHAR-2086:
-------------------------------------

Sunil Parmar from ThreatMatrix said that they are interested in the 0.9 Exactly once operator.

> Kafka Output Operator with Kafka 0.9 API
> ----------------------------------------
>
>                 Key: APEXMALHAR-2086
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2086
>             Project: Apache Apex Malhar
>          Issue Type: New Feature
>            Reporter: Sandesh
>            Assignee: Sandesh
>
> Goal : 2 Operartors for Kafka Output
>       1. Simple Kafka Output Operator 
>             - Supports Atleast Once 
>             - Expose most used producer properties as class properties
>       2. Exactly Once Kafka Output ( Not possible in all the cases, will be documented later )
>             
> Design for Exactly Once
> Window Data Manager - Stores the Kafka partitions offsets.
> Kafka Key - Used by the operator = AppID#OperatorId
> During recovery. Partially written window is re-created using the following  approach:
> Tuples between the largest recovery offsets and the current offset are checked. Based on the key, tuples written by the other entities are discarded. 
> Only tuples which are not in the recovered set are emitted.
> Tuples needs to be unique within the window.
>       


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)