lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Max Pfingsthorn" <m.pfingsth...@hippo.nl>
Subject RE: Optimize completely in memory with a FSDirectory?
Date Thu, 06 Apr 2006 09:25:25 GMT
Hi,

Thanks for your suggestion. I thought about the same, but somehow it didn't seem like such
a good idea... Now that I think about it, it would take the same IO load (in terms of flushing
many megabytes to disk) as optimizing in memory with the FSDirectory.

Another weird thing we observed here is this:

During incremental updates to a previously optimized index, no matter what I set the merge
factor at, it seems to optimize or possibly merge much sooner than it should.
More clearly:

I have an optimized index of around 150MB. I set merge factor to 300, maxmergedocs to Interger.MAX_VALUE,
minmergedocs (maxbuffereddocs) to 50000 (I have 40000 docs in the index), and still it merges
after around 50-80 new documents. If I understand merge factor right, it should not merge
at all, but start a new segment after 300 new documents.

Of course this is a very artificial set of parameters, but I wanted to see what goes on. Could
it have anything to do with the fact that I close the indexwriter after each batch of updates?
Can anyone explain this?

max

> -----Original Message-----
> From: Daniel Naber [mailto:lucenelist2005@danielnaber.de]
> Sent: Wednesday, April 05, 2006 20:23
> To: java-user@lucene.apache.org
> Subject: Re: Optimize completely in memory with a FSDirectory?
> 
> 
> On Mittwoch 05 April 2006 13:02, Max Pfingsthorn wrote:
> 
> > The setMaxBufferedDocs and related parameters help a lot already to
> > fully exploit my RAM when indexing, but since I'm running a 
> fairly small
> > index of around 40000 docs and I'm optimizing it relatively 
> often, I was
> > wondering if there is any way to enforce complete in-memory
> > optimization.
> 
> Maybe you could use a RAMDirectory and write it to disk using 
> IndexWriter.addIndexes() from time to time?
> 
> Regards
>  Daniel
> 
> -- 
> http://www.danielnaber.de
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message