lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian Doyle" <brianpdo...@gmail.com>
Subject custom sort and out of memory
Date Mon, 18 Feb 2008 01:52:39 GMT
We've written our own custom sorter to be able to sort on the latitude and
longitude fields from the results.   We have an index that is about
18million records and 12GB on disk in size.  We allocated about 3GB of heap
to the index and with about 1 request to the index every 2 or 3 seconds we
would run out of memory about once every 1.5 hours.   We modified our custom
sort comparators to implement the equals and hashcode methods and used a
WeakHashMap to cache the doc ids to and their lat/lon values. We ran some
tests and it started to reuse those comparators and now it will go for maybe
6 to 9 hours before running out of memory, however, it's still running out
of memory.

The only other object that isn't in our sorter, that isn't in the
WeakHashMap is an instance of the index reader. We use that reader as
instance variable in order to get the latitude and longitude values to sort
on.   Could having a reference to the index reader be causing the memory
leak?   We were considering making the instance of the index reader a
WeakReference.    When we did some profiling we noticed that before it runs
out of memory and the org.apache.lucene.search.FieldSortedHitQueue class was
taking up 331MB and the org.apache.lucene.index.IndexReader[23] class was
taking up 111MB of memory.   The [23] means it there were 23 instances.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message