apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bright chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2361) Optimise SpillableWindowedKeyedStorage remove(Window) to improve the performance
Date Thu, 01 Dec 2016 00:15:58 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15710303#comment-15710303
] 

bright chen commented on APEXMALHAR-2361:
-----------------------------------------

Probably we can handle DISCARDING by update value to default value instead of clear the window.
Then  remove(Window)  only used for window after lateness.

> Optimise SpillableWindowedKeyedStorage remove(Window) to improve the performance
> --------------------------------------------------------------------------------
>
>                 Key: APEXMALHAR-2361
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2361
>             Project: Apache Apex Malhar
>          Issue Type: Improvement
>            Reporter: bright chen
>            Assignee: bright chen
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> Currently, SpillableWindowedKeyedStorage remove(Window) will go through each key and
mark all of them as deleted. It would be expensive when there are lots of keys and especially
these entry already spill out of memory (this the common case when remove() was called).
> Suggest to mark whole window as deleted. When the window was marked as deleted, it will
not allowed to add/update any entry of this window ( this should match the requirement as
remove(Window) only be called after allowed lateness



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message