lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: deprecated optimize()!
Date Fri, 27 Jan 2012 23:35:39 GMT
Hi,

> After reading all about the renaming of optimize() and updating my Lucene
> libraries to 3.4, I was surprised and confused by what I found.
> 
> I have a 1 segment  index (all files are named _1*.*)  that had been
created
> with 3.0.1 code which had been optimized many times (all 3.0.1 code).  The
> first time my code used the 3.4 libraries with version level set to 3.4
and it tried
> to optimize() (still using this now deprecated old call), the new code
went wild!
> It took up more memory than the heap was limited to, so I believe it is
taking
> up system resources.   We have turned off optimize for now.

Did it throw OutOfMemoryException? I assume not - but I assume, you have
seen more virtual memory usage in "top", but that's not really related to
optimize/forceMerge. If you are running on a 64 bit platform, Lucene 3.4
will default to MMapDirectory on FSDirectory.open(), which maps the index
like a linux/windows swap file into *virtual* memory (not necessary physical
memory, but it then directly accesses the file system cache instead of
copying to Java heap). This is why "top" mem usage grows, even multiple
times of java heap. Nothing to worry about - that's expected and normal (see
http://en.wikipedia.org/wiki/Mmap)!

> What is it doing?  (I believe it is using a TieredMergePolicy, see the
IndexWriter
> constructor).

The merge policy does not have anything to do with that, optimizing or
forcemerge to one index segment is the same in all merge policies.

> Anyone have any resources to point me to on the subject so I can pick the
right
> call to forceMerge() and a good MergePolicy before upgrading to Lucene
4.0?
> Any ideas?    What should be a reasonable 3.4 approach?   Are there issues
> about the the just complete IndexWriter?  Ho about open readers (in the
web
> app using the index).

If you have open readers, many files are open and mapped into virtual
memory, so virtual memory usage grows.

> -Paul
> 
> p.s. More details available upon request.
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message