apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2335) Problems on StateTracker
Date Tue, 15 Nov 2016 02:10:58 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665759#comment-15665759
] 

ASF GitHub Bot commented on APEXMALHAR-2335:
--------------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/apex-malhar/pull/493


> Problems on StateTracker
> ------------------------
>
>                 Key: APEXMALHAR-2335
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2335
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: bright chen
>            Assignee: bright chen
>   Original Estimate: 144h
>  Remaining Estimate: 144h
>
> The intension of StateTracker#bucketAccessed as I understand was to keep the last access
time of each bucket so freeMemory can free the buckets which not accessed recently.
> But here are some problems
> - StateTracker#bucketAccessed() will be called each time access to the bucket. Current
implementation use Concurrent map and set to handle it(bucketHeap and bucketAccessTimes).
it could be pretty heavy.
> - StateTracker#run use bucketHeap to get the bucket which need to free memory, but at
the same time bucketHeap was constantly changed by bucketAccessed(). It could cause concurrent
issue event bucketHeap is a Current set. And even more the while loop could never end. Following
two bug probably related to this:
> https://issues.apache.org/jira/browse/APEXMALHAR-2333
> https://issues.apache.org/jira/browse/APEXMALHAR-2334
> Following are my thought to solve these issue.
> - It's not necessary to keep very accurate of update time, it probably only need to be
updated once for each stream window. But StateTracker don't support endWindow() (in fact endWindow()
was triggered at end of application window). So, we can keep the bucket which updated in this
period and update only at the end of the period.
> - keep two sets of bucketHeap, one for bucketAccessed() and one for run(). so the lock
only need when switch



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message