apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "bright chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (APEXMALHAR-2332) StateTracker should free memory after committed
Date Tue, 08 Nov 2016 23:35:59 GMT

     [ https://issues.apache.org/jira/browse/APEXMALHAR-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

bright chen updated APEXMALHAR-2332:
------------------------------------
    Remaining Estimate: 120h  (was: 48h)
     Original Estimate: 120h  (was: 48h)
           Description: 
Current StateTracker free memory was triggered by a timer. The default the timer value was
DAGContext.STREAMING_WINDOW_SIZE_MILLIS.defaultValue * OperatorContext.APPLICATION_WINDOW_COUNT.defaultValue.
It would have memory leak if the process with operator thread and memory release thread as
following:

bucket1: put(), put() ... put()
bucket2: put(), put() ... put()
freeMemory(): {bucket removed from bucketHeap and bucketAccessTimes}
commit: bucket1, bucket2

in this case,  nothing was freed and the bucket can't be freed any more

And the default value of free memory could large and memory used up even before get chance
of free memory. 

  was:
StateTracker#bucketAccessTimes keep the bucket access time order by access time. It was used
by free memory thread to decide which bucket can be freed. So, each access to bucket include
put and get should update the access time.

As bucketAccessTimes and bucketHeap are shared by two thread. update them for each operation
could impact the performance. It better to update period. As Bucket don't support window operation.
I am going to keep the update time and update when time out.


               Summary: StateTracker should free memory after committed  (was: StateTracker#bucketAccessed
should be called each time access to the bucket)

> StateTracker should free memory after committed
> -----------------------------------------------
>
>                 Key: APEXMALHAR-2332
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2332
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: bright chen
>            Assignee: bright chen
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> Current StateTracker free memory was triggered by a timer. The default the timer value
was DAGContext.STREAMING_WINDOW_SIZE_MILLIS.defaultValue * OperatorContext.APPLICATION_WINDOW_COUNT.defaultValue.
It would have memory leak if the process with operator thread and memory release thread as
following:
> bucket1: put(), put() ... put()
> bucket2: put(), put() ... put()
> freeMemory(): {bucket removed from bucketHeap and bucketAccessTimes}
> commit: bucket1, bucket2
> in this case,  nothing was freed and the bucket can't be freed any more
> And the default value of free memory could large and memory used up even before get chance
of free memory. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message