lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From robert engels <reng...@ix.netcom.com>
Subject Re: code improvement / easier optimization
Date Fri, 02 Nov 2007 18:51:42 GMT
To clarify, I changed the code in IndexWriter that creates the  
SegmentReaders DURING MERGING to use...

On Nov 2, 2007, at 1:40 PM, robert engels wrote:

> The Lucene 2.2 code for managing buffers is somewhat "ugly" - the  
> passing of the buffer size parameter around.
>
> I changed this in my branch to use the BufferSizes class below.
>
> I changed the BufferedIndexInput/Output class like this
>
> class BufferedIndexInput {
> 	private int bufferSize = BufferSizes.getReadBufferSize();
> 	....
> }
> class BufferedIndexOutput {
> 	private int bufferSize = BufferSizes.getWriteBufferSize();
> 	....
> }
>
> then in IndexWriter I surround the code that creates the  
> SegmentReaders with:
>
> try {
>      BufferSizes.useMergeBuffers();
>      ... create segment readers ...
> } finally {
>      BufferSizes.useNormalBuffers();
> }
>
> I think this is much cleaner. It also allows for other  
> optimizations like:
>
> query engine detects a phrase query, so it increase the the buffers  
> prior to reading the terms
> query result has a lot of matches, so increase the buffer size when  
> reading the documents
>
> Seems a lot easier to manage. It also allows playing with various  
> buffer sizes very easily.
>
> I have been able to get the optimize time down from 3.5 minutes to  
> 1.5 minutes on the exact same index (using all of the recent  
> enhancements - much of the improvement is related to the larger  
> buffer sizes used in Lucene 2.2).
>
>
> package org.apache.lucene.store;
>
> public class BufferSizes {
>     private static ThreadLocal useMergeBuffers = new ThreadLocal(){};
>
>     public static int getReadBufferSize() {
> 	return (Boolean.TRUE.equals(useMergeBuffers.get())) ? 16384*2 : 1024;
>     }
>     public static int getWriteBufferSize() {
> 	return 16384*2;
>     }
>     /**
>      * cause the current thread to use buffers sized for segment  
> merging. always use try/finally to reset the value
>      */
>     public static void useMergeBuffers() {
> 	useMergeBuffers.set(Boolean.TRUE);
>     }
>     /**
>      * cause the current thread to use buffers sized for normal  
> index operations
>      */
>     public static void useNormalBuffers() {
> 	useMergeBuffers.set(Boolean.FALSE);
>     }
> }
>
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message