asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Carey <dtab...@gmail.com>
Subject Re: Issue 910 in asterixdb: Index creation/use can cause resource leakage
Date Wed, 22 Jul 2015 23:30:08 GMT
Ah.  The "leak" I am thinking of is more an unreleased use of memory - 
we should make sure that we activate AND deactivate indexes eventually 
and don't just have more and more in-memory state over time.  We should 
probably do a "lifecycle review" of how we manage (i) metadata cache 
entries, (ii) index metadata, and (iii) in-memory index components - 
anything that has a memory footprint at runtime - and make sure we know 
when each first hits memory and then UN-hits memory at some reasonable 
time.  We should also make sure that not all indexes have in-memory 
components; if we are only reading them we shouldn't need those.  
Perhaps we could do this in a "crowd-sourced" Google doc that identifies 
the key lifecycle events (create index, access dataset for first time, 
access index of dataset for first time, hit limit on number of active 
datasets, ....).

On 7/22/15 12:58 AM, Ian Maxon wrote:
> Index open/close is probably fine. I haven't traced it in a debugger
> or anything, but I believe it would just look up the index in the map
> rather than creating a new object. The only way to create a leak is to
> create many unique indexes within an NC's lifetime.
>
> On Wed, Jul 22, 2015 at 12:39 AM, Mike Carey <dtabass@gmail.com> wrote:
>> But my guess is that there's similar unfortunate behavior related to index
>> open and close...?
>> (Which would happen way more regularly?)  Agreed w.r.t. priority if we are
>> in "prototype" mode,
>> but if anyone wanted to go "live" and run for months at a time, this would
>> be a leak we'd feel,
>> potentially.  We need to devise and do some testing around index/memory
>> management and
>> the lifecycle there.  :-)
>>
>> On 7/22/15 12:25 AM, abdullah alamoudi wrote:
>>> It should be fixed but I don't think that it is a release blocker since
>>> this rarely happen and the amount of memory per index instance object is
>>> almost nothing.
>>>
>>> just my 2 cents.
>>>
>>> On Wed, Jul 22, 2015 at 10:10 AM, Michael Carey <mjcarey@ics.uci.edu>
>>> wrote:
>>>
>>>> Definitely a high priority bug - we should try to fix this pre-release!
>>>>
>>>>
>>>> -------- Forwarded Message --------
>>>> Subject:        Issue 910 in asterixdb: Index creation/use can cause
>>>> resource leakage
>>>> Date:   Wed, 22 Jul 2015 03:13:59 +0000
>>>> From:   asterixdb@googlecode.com
>>>> Reply-To:       asterixdb-dev@googlegroups.com
>>>> To:     asterixdb-dev@googlegroups.com
>>>>
>>>>
>>>>
>>>> Status: Accepted
>>>> Owner: ima...@uci.edu
>>>> CC: buyingyi@gmail.com
>>>> Labels: Type-Defect Priority-High
>>>>
>>>> New issue 910 by ima...@uci.edu: Index creation/use can cause resource
>>>> leakage
>>>> https://code.google.com/p/asterixdb/issues/detail?id=910
>>>>
>>>> In IndexDataflowHelper, calling create() will make an index instance, and
>>>> create it. This inserts an entry into the IndexLifecycleManager's
>>>> resource
>>>> map. Entries in this map are only freed once unregister() is called on an
>>>> index. However this only happens when an index is destroyed, not when it
>>>> is
>>>> closed. There is no other way in which references to indexes are garbage
>>>> collected (even though there is a reference counter in the IndexInfo).
>>>> Therefore creating or using an index uses resources in this map even if
>>>> they are not active, and were created during the lifetime of the NC- and
>>>> there is no way to free that memory.
>>>>
>>>> --
>>>> You received this message because this project is configured to send all
>>>> issue notifications to this address.
>>>> You may adjust your notification preferences at:
>>>> https://code.google.com/hosting/settings
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "asterixdb-dev" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to asterixdb-dev+unsubscribe@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
>>>>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message