lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Artem Vasiliev" <>
Subject Re: Out of memory exception for big indexes
Date Tue, 24 Apr 2007 13:16:39 GMT
Hello Ivan!

It's so sad to me that you had bad results with that patch. :)

The discussion in the ticket is out-of-date - the patch was initially in
several classes, used WeakHashMap but then it evolved to what it's now - one
StoredFieldSortFactory class. I use it in my sharehound app in pretty much
the same the form it is in Jira currently and it does show good results to

In your sample searches,
- how many results do you have?
- how long does the sorted search execute?
- what is the average size of a sorted field?
- what is the CPU and how much of it and memory you give to the application?

I get page 1 (first 100 items) of sorted list with 10000 items in 0.3s to 3s
(for date column it exactly depends on whether the sort is ascending or
descending - don't know why is that). My index is about 1mln docs and 1G;
sorted fields are rather small (numbers, dates and string of maybe 50
symbols average). The machine looks quite beefy to me - Intel core duo with
500M given to the application.


On 4/23/07, Ivan Vasilev <> wrote:
> Hi All,
> I put this problem in the forum but I had no chance to work on it last
> week unfurtunately...
> So now I tested the Artem's patch but the results show:
> 1) speed is very slow compare with the usage without patch
> 2) There are not very big differences of memory usage (I tested till now
> only with relativly small indexes - less than 1 GB and less than 1 mil
> docs because the when using with 20-40 GB indexes I had to wait more
> than 5 mins what is practically usless).
> So I have doubts if I use the patch correctly. I do just what is
> described in Artem's letter:
> AV> You can include StoredFieldSortFactory class source file into your
> sources and
> AV> then use StoredFieldSortFactory.create(sortFieldName, sortDescending)
> to get
> AV> Sort object for sorting query.
> AV> StoredFieldSortFactory source file can be extracted from LUCENE-769
> patch or
> AV> from sharehound sources:
> What I am wondering about is that in the patch commetns
> ( I see that there is
> written that patch solves the problem by using WeakHashMap, but actually
> in the downloaded  file there is not used
> WeakHashMap. Another thing: In the comments in Lucene-769 issue there is
> mentioned something about classes like: WeakDocumentsCache and
> DocCachingIndexReader but I did not found them in Lucene source code
> neither as classes in So my questions are:
> 1. Is it enought to include the file in the
> source code or there are also other classes that I have to douwnload and
> include?
> 2. Have I to use this DocCachingIndexReader instead of Reader that I
> currently use in cases when I expect OOMException and will use this patch?
> Thanks to all once again :),
> Ivan
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message