apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2335) Problems on StateTracker
Date Thu, 10 Nov 2016 23:01:03 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655462#comment-15655462

ASF GitHub Bot commented on APEXMALHAR-2335:

GitHub user brightchen opened a pull request:


    APEXMALHAR-2335 APEXMALHAR-2333 APEXMALHAR-2334 #resolve #comment Pro…

    …blems on StateTracker

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/brightchen/apex-malhar APEXMALHAR-2335-PR

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #493
commit 8fd0574df26f9191b5a69d84fd8bddbf3214a33a
Author: brightchen <bright@datatorrent.com>
Date:   2016-11-10T22:59:52Z

    APEXMALHAR-2335 APEXMALHAR-2333 APEXMALHAR-2334 #resolve #comment Problems on StateTracker


> Problems on StateTracker
> ------------------------
>                 Key: APEXMALHAR-2335
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2335
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: bright chen
>            Assignee: bright chen
>   Original Estimate: 144h
>  Remaining Estimate: 144h
> The intension of StateTracker#bucketAccessed as I understand was to keep the last access
time of each bucket so freeMemory can free the buckets which not accessed recently.
> But here are some problems
> - StateTracker#bucketAccessed() will be called each time access to the bucket. Current
implementation use Concurrent map and set to handle it(bucketHeap and bucketAccessTimes).
it could be pretty heavy.
> - StateTracker#run use bucketHeap to get the bucket which need to free memory, but at
the same time bucketHeap was constantly changed by bucketAccessed(). It could cause concurrent
issue event bucketHeap is a Current set. And even more the while loop could never end. Following
two bug probably related to this:
> https://issues.apache.org/jira/browse/APEXMALHAR-2333
> https://issues.apache.org/jira/browse/APEXMALHAR-2334
> Following are my thought to solve these issue.
> - It's not necessary to keep very accurate of update time, it probably only need to be
updated once for each stream window. But StateTracker don't support endWindow() (in fact endWindow()
was triggered at end of application window). So, we can keep the bucket which updated in this
period and update only at the end of the period.
> - keep two sets of bucketHeap, one for bucketAccessed() and one for run(). so the lock
only need when switch

This message was sent by Atlassian JIRA

View raw message