lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vsevel <v.se...@lombardodier.com>
Subject Re: Searching while optimizing
Date Tue, 24 Nov 2009 06:44:32 GMT

1) correct: I am using IndexWriter.getReader(). I guess I was assuming that
was a privately owned object and I had no business dealing with its
lifecycle. the api would be clearer to rename the operation createReader().

2) how much transient disk space should I expect? isn't this pretty much
what the index writer javadoc said we should not do: "When running in this
mode, be careful not to refresh your readers while optimize or segment
merges are taking place as this can tie up substantial disk space."


Michael McCandless-2 wrote:
> 
> When you say "getting a reader of the writer" do you mean
> writer.getReader()?  Ie the new near real-time API in 2.9?
> 
> For that API (an in general whenever you open a reader), you must
> close it.  I think all your files is because you're not closing your
> old readers.
> 
> Reopening readers during optimize is fine, if you close the old reader
> each time.  It will possibly tie up more transient disk usage than had
> you reopened at the end of optimize, but if you have plenty of disk
> space it shouldn't be a problem.
> 
> Mike
> 
> On Mon, Nov 23, 2009 at 3:20 PM, vsevel <v.sevel@lombardodier.com> wrote:
>>
>> Hi, I am using lucene 2.9.1 to index a continuous flow of events. My
>> server
>> keeps an index writer open at all time and write events as groups of a
>> few
>> hundred followed by a commit. While writing, users invoke my server to
>> perform searches. Once a day I optimize the index, while writes happens
>> and
>> searches may happen. I adopted the following strategy:
>>
>> for every search I open a new IndexSearcher of the reader of the writer.
>> I
>> execute the search, fetch the documents and finally close the searcher.
>> Specifically, I never close the reader, nor the writer.
>>
>> Q: is that a reasonnable strategy?
>>
>> I found out that my 40Gb index grew up to 200Gb while the number of docs
>> stayed put at 30 millions. I am suspecting that a search during the
>> optimize
>> caused this situation, as described in the index writer javadoc (about
>> refreshing readers during an optimize).
>>
>> Q: is that the likely cause? is getting a reader of the writer just as
>> "bad"
>> as refreshing a reader during an optimize? how can I avoid this behavior?
>> should I just deny searches while optimizing?
>>
>> question on the side: is there any way to interrupt a search that takes
>> too
>> long? for instance by setting a boolean from another thread on the
>> searcher
>> currently performing the search.
>>
>> thanks,
>> vincent
>> --
>> View this message in context:
>> http://old.nabble.com/Searching-while-optimizing-tp26485138p26485138.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/Searching-while-optimizing-tp26485138p26491155.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message