apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay M Pujare (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2403) Finalization is not invoked for AbstractFileOutputOperator<INPUT>
Date Thu, 09 Feb 2017 01:58:41 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858851#comment-15858851

Sanjay M Pujare commented on APEXMALHAR-2403:

That is or cannot be done here. The user has decided to shut down the application (thru Apex
CLI) because he has noticed that the app has finished reading the whole SQL table and has
determined that no more records are going to be added. Shouldn't the shutdown command tell
the FileOutoutOperator to finalize the files i.e. rename from temp to actual names? This is
an example of a batch application where the completion of batch is determined externally (not
thru EOF on the input) and it will be good to support such use cases.

> Finalization is not invoked for AbstractFileOutputOperator<INPUT>
> -----------------------------------------------------------------
>                 Key: APEXMALHAR-2403
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2403
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: Sanjay M Pujare
>            Priority: Minor
> I used an example/template program for Database to HDFS data copy. While copying the
table, I see "temp" files created on the output HDFS side which I expected to be "finalized"
to non-temp names when the app is shut down. But that did not happen. AbstractFileOutputOperator<INPUT>
and its subclasses should do proper finalization via the deactivate() method when an app is
properly shut down.

This message was sent by Atlassian JIRA

View raw message