lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Busch <busch...@gmail.com>
Subject Re: Dynamically varying maxBufferedDocs
Date Thu, 09 Nov 2006 19:13:09 GMT
I had the same problem with large documents causing memory problems. I 
solved this problem by introducing a new setting in IndexWriter 
setMaxBufferSize(long). Now a merge is either triggered when 
bufferedDocs==maxBufferedDocs *or* the size of the bufferedDocs >= 
maxBufferSize. I made these changes based on the new merge policy Yonik 
mentioned, so if anyone is interested I could open a Jira issue and 
submit a patch.

- Michael


Yonik Seeley wrote:
> On 11/9/06, Chuck Williams <chuck@manawiz.com> wrote:
>> Thanks Yonik!  Poor wording on my part.  I won't vary maxBufferedDocs,
>> just am making flushRamSegments() public and calling it externally
>> (properly synchronized), earlier than it would otherwise be called from
>> ongoing addDocument-driven merging.
>>
>> Sounds like this should work.
>
> Yep.
> For best behavior, you probably want to be using the current
> (svn-trunk) version of Lucene with the new merge policy.  It ensures
> there are mergeFactor segments with size <= maxBufferedDocs before
> triggering a merge.  This makes for faster indexing in the presence of
> deleted docs or partially full segments.
>
> -Yonik
> http://incubator.apache.org/solr Solr, the open-source Lucene search 
> server
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message