lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] Created: (LUCENE-2357) Reduce transient RAM usage while merging by using packed ints array for docID re-mapping
Date Mon, 29 Mar 2010 11:21:27 GMT
Reduce transient RAM usage while merging by using packed ints array for docID re-mapping
----------------------------------------------------------------------------------------

                 Key: LUCENE-2357
                 URL: https://issues.apache.org/jira/browse/LUCENE-2357
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
            Reporter: Michael McCandless
            Priority: Minor
             Fix For: 3.1


We allocate this int[] to remap docIDs due to compaction of deleted ones.

This uses alot of RAM for large segment merges, and can fail to allocate due to fragmentation
on 32 bit JREs.

Now that we have packed ints, a simple fix would be to use a packed int array... and maybe
instead of storing abs docID in the mapping, we could store the number of del docs seen so
far (so the remap would do a lookup then a subtract).  This may add some CPU cost to merging
but should bring down transient RAM usage quite a bit.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message