apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From PramodSSImmaneni <...@git.apache.org>
Subject [GitHub] incubator-apex-core pull request: Skipping endWindow and operator ...
Date Thu, 29 Oct 2015 14:27:06 GMT
Github user PramodSSImmaneni commented on the pull request:

    https://github.com/apache/incubator-apex-core/pull/143#issuecomment-152196716
  
    We talked about this. This is causing a bug right now with the operators. Many operators
assume that the endWindow callback is always called at the end of processing all the tuples
in the window and do some persistent processing, e.g., database operators commit transactions
in endWindow and ignore the data for those windows in replay and would result in wrong results,
idempotent operators write persistent state in endWindow. Immaterial of what the original
intent was it is currently being interpreted as being called after all tuples in window are
processed. Hence the change. There will be additional work to notify the operator via a listener
callback if it was terminated prematurely by the system. That is coming. Doing it as a listener
avoids having to change endWindow semantics and cause backwards incompatibility.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message