lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Busch <busch...@gmail.com>
Subject Re: [jira] Resolved: (LUCENE-709) [PATCH] Enable application-level management of IndexWriter.ramDirectory size
Date Wed, 22 Nov 2006 18:47:47 GMT
Ning Li wrote:
> I was away so I'm catching up.
>
> If this (occasional large documents consume too much memory) happens
> to a few applications, should it be solved in IndexWriter?
>
> A possible design could be:
> First, in addDocument(), compute the byte size of a ram segment after
> the ram segment is created. In the synchronized block, when the newly
> created segment is added to ramSegmentInfos, also add its byte size to
> the total byte size of ram segments.
> Then, in maybeFlushRamSegments(), either one of two conditions can
> trigger a flush: number of ram segments reaching maxBufferedDocs, and
> total byte size of ram segments exceeding a threshold.
>
> The overhead is very small in this design. Of course, IndexWriter
> would have another configurable parameter. :-) But it's nice if an
> application could set a limit on the memory it uses to buffer docs.
>
> Ning
>

This is exactly how I implemented it in my private version a couple of 
weeks ago. It works good and I don't see performance problems with this 
design. I named the new parameter in IndexWriter: setMaxBufferSize(long).

Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message