lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (LUCENE-3425) NRT Caching Dir to allow for exact memory usage, better buffer allocation and "global" cross indices control
Date Thu, 09 May 2013 23:05:45 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Uwe Schindler updated LUCENE-3425:
----------------------------------

    Fix Version/s:     (was: 4.3)
                   4.4
    
> NRT Caching Dir to allow for exact memory usage, better buffer allocation and "global"
cross indices control
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3425
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3425
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.4, 4.0-ALPHA
>            Reporter: Shay Banon
>             Fix For: 4.4
>
>
> A discussion on IRC raised several improvements that can be made to NRT caching dir.
Some of the problems it currently has are:
> 1. Not explicitly controlling the memory usage, which can result in overusing memory
(for example, large new segments being committed because refreshing is too far behind).
> 2. Heap fragmentation because of constant allocation of (probably promoted to old gen)
byte buffers.
> 3. Not being able to control the memory usage across indices for multi index usage within
a single JVM.
> A suggested solution (which still needs to be ironed out) is to have a BufferAllocator
that controls allocation of byte[], and allow to return unused byte[] to it. It will have
a cap on the size of memory it allows to be allocated.
> The NRT caching dir will use the allocator, which can either be provided (for usage across
several indices) or created internally. The caching dir will also create a wrapped IndexOutput,
that will flush to the main dir if the allocator can no longer provide byte[] (exhausted).
> When a file is "flushed" from the cache to the main directory, it will return all the
currently allocated byte[] to the BufferAllocator to be reused by other "files".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message