Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
Date: Tue, 4 Oct 2016 18:45:20 +0000 (UTC)
From: "Munagala V. Ramanath (JIRA)" <jira@apache.org>
To: dev@apex.incubator.apache.org
Message-ID: <JIRA.13006348.1474419226000.733862.1475606720739@Atlassian.JIRA>
In-Reply-To: <JIRA.13006348.1474419226000@Atlassian.JIRA>
References: <JIRA.13006348.1474419226000@Atlassian.JIRA> <JIRA.13006348.1474419226387@arcas>
Subject: [jira] [Comment Edited] (APEXMALHAR-2254) File input operator is
 not idempotent with closing files on replay
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 04 Oct 2016 18:45:30 -0000


    [ https://issues.apache.org/jira/browse/APEXMALHAR-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15546182#comment-15546182 ] 

Munagala V. Ramanath edited comment on APEXMALHAR-2254 at 10/4/16 6:45 PM:
---------------------------------------------------------------------------

It would be useful to define what we mean for this operator (AbstractFileInputOperator) to be idempotent: It doesn't quite work to define it as "same tuples output in the same windows, given same input" because this operator is abstract, has no ports and emits no tuples.

Concrete subclasses are expected to implement the emit() method to actually emit tuples. So a definition that involves calls to emit(), openFile(), closeFile() may be more meaningful. Also, clarifying what is expected of subclasses in order to maintain idempotency will also be useful.

Finally, there are several other JIRAs related to this class and it may be useful to keep them in mind too.


was (Author: dtram):
It would be useful to define what we mean for this operator (AbstractFileInputOperator) to be idempotent: It doesn't quite to define it as "same tuples output in the same windows, given same input" because this operator is abstract, has no ports and emits no tuples.

Concrete subclasses are expected to implement the emit() method to actually emit tuples. So a definition that involves calls to emit(), openFile(), closeFile() may be more meaningful. Also, clarifying what is expected of subclasses in order to maintain idempotency will also be useful.

Finally, there are several other JIRAs related to this class and it may be useful to keep them in mind too.

> File input operator is not idempotent with closing files on replay
> ------------------------------------------------------------------
>
>                 Key: APEXMALHAR-2254
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2254
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: Pramod Immaneni
>            Assignee: Pramod Immaneni
>
> With the file input operator, on a replay in a failure scenario, the same data is output as before the failure, for every window that is being replayed after checkpoint. To do this the operator keeps track of the files and offsets for every window and replays the data based on that. 
> However, if it so happens that before the failure the processing of a file was finished and it was closed exactly before the end window and the next file was opened and processed in a new window, in the replay the closing of the first file does not happen in earlier window but happens in the latter window. This can cause problems if an operator depends on the closing file also to happen in an idempotent manner.
> Improve the operator to save the closing and opening of files in the idempotent state as well so that it can also happen in an idempotent manner.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)