apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pramod Immaneni (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (APEXMALHAR-2017) Use pre checkpoint notification to optimize operator IO
Date Mon, 12 Sep 2016 18:35:21 GMT

     [ https://issues.apache.org/jira/browse/APEXMALHAR-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pramod Immaneni updated APEXMALHAR-2017:
----------------------------------------
    Assignee: Velineni Lakshmi Prasanna  (was: Pramod Immaneni)

> Use pre checkpoint notification to optimize operator IO
> -------------------------------------------------------
>
>                 Key: APEXMALHAR-2017
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2017
>             Project: Apache Apex Malhar
>          Issue Type: Improvement
>            Reporter: Pramod Immaneni
>            Assignee: Velineni Lakshmi Prasanna
>
> Currently many output operators enforce persistence of data on endWindow by calling flush,
hflush or equivalent calls. This was done to help recovery. Doing this always ensures that
the data corresponding to checkpoint state at recovery is always present.
> A recent addition to the engine lets the operators know about an impending checkpoint
just before it happens using a callback. Operators can now enforce persistence of data one
time in this in this callback instead of end of every window. This results in better performance
as data is not being frequently written to persistent storage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message