lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: [jira] Commented: (LUCENE-1043) Speedup merging of stored fields when field mapping "matches"
Date Fri, 02 Nov 2007 20:49:07 GMT
I am working on a bulk document copy right now - I will let you know  
if it improves things much.

I doubt it, because I already configure the streams to use fairly  
large input and output buffers during a merge - but the memory index  
merge may see additional benefits due to lower CPU calls.


On Nov 2, 2007, at 3:39 PM, robert engels (JIRA) wrote:

>
>     [ https://issues.apache.org/jira/browse/LUCENE-1043? 
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment- 
> tabpanel#action_12539696 ]
>
> robert engels commented on LUCENE-1043:
> ---------------------------------------
>
> When bulk copying the documents, I think you need to:
>
> read array of long from index (8 * (ndocs+1)) in long[ndocs+1]  
> offsets;
> calculate length = offset[ndocs]-offset[0];
> read bytes of length from document file
> startoffset = current output document stream position
> write bytes to output document
> modify offset[] adding startoffset-offset[0] to each entry
> write offset[] in bulk to index output
>
>> Speedup merging of stored fields when field mapping "matches"
>> -------------------------------------------------------------
>>
>>                 Key: LUCENE-1043
>>                 URL: https://issues.apache.org/jira/browse/ 
>> LUCENE-1043
>>             Project: Lucene - Java
>>          Issue Type: Improvement
>>          Components: Index
>>    Affects Versions: 2.2
>>            Reporter: Michael McCandless
>>            Assignee: Michael McCandless
>>            Priority: Minor
>>             Fix For: 2.3
>>
>>         Attachments: LUCENE-1043.patch
>>
>>
>> Robert Engels suggested the following idea, here:
>>   http://www.gossamer-threads.com/lists/lucene/java-dev/54217
>> When merging in the stored fields from a segment, if the field  
>> name ->
>> number mapping is identical then we can simply bulk copy the entire
>> entry for the document rather than re-interpreting and then re- 
>> writing
>> the actual stored fields.
>> I've pulled the code from the above thread and got it working on the
>> current trunk.
>
> -- 
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message