lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Artem Vasiliev" <abub...@gmail.com>
Subject Re: Out of memory exception for big indexes
Date Tue, 24 Apr 2007 13:29:42 GMT
Hi Ivan!

btw may be forbidding the sorted search in case of too many results is an
option? I did this way in my case.

Regards,
Artem.

On 4/24/07, Artem Vasiliev <abublic@gmail.com> wrote:
>
> Ahhh, you said in your original post that your search matches _all_ the
> results.. Yup my patch will not help much in this case - after all all the
> values have to be read to be compared while sorting! :)
>
> LUCENE-769 patch helps only if result set is significantly less than full
> index size.
>
> Regards,
> Artem
>
> On 4/24/07, Artem Vasiliev <abublic@gmail.com> wrote:
> >
> > Hello Ivan!
> >
> > It's so sad to me that you had bad results with that patch. :)
> >
> > The discussion in the ticket is out-of-date - the patch was initially in
> > several classes, used WeakHashMap but then it evolved to what it's now - one
> > StoredFieldSortFactory class. I use it in my sharehound app in pretty much
> > the same the form it is in Jira currently and it does show good results to
> > me.
> >
> > In your sample searches,
> > - how many results do you have?
> > - how long does the sorted search execute?
> > - what is the average size of a sorted field?
> > - what is the CPU and how much of it and memory you give to the
> > application?
> >
> > I get page 1 (first 100 items) of sorted list with 10000 items in 0.3sto 3s (for
date column it exactly depends on whether the sort is ascending
> > or descending - don't know why is that). My index is about 1mln docs and 1G;
> > sorted fields are rather small (numbers, dates and string of maybe 50
> > symbols average). The machine looks quite beefy to me - Intel core duo with
> > 500M given to the application.
> >
> > Regards,
> > Artem
> >
> > On 4/23/07, Ivan Vasilev < ivasilev@sirma.bg> wrote:
> > >
> > > Hi All,
> > > THANK YOU FOR YOUR HELP :)
> > > I put this problem in the forum but I had no chance to work on it last
> > > week unfurtunately...
> > > So now I tested the Artem's patch but the results show:
> > > 1) speed is very slow compare with the usage without patch
> > > 2) There are not very big differences of memory usage (I tested till
> > > now
> > > only with relativly small indexes - less than 1 GB and less than 1 mil
> > > docs because the when using with 20-40 GB indexes I had to wait more
> > > than 5 mins what is practically usless).
> > >
> > > So I have doubts if I use the patch correctly. I do just what is
> > > described in Artem's letter:
> > >
> > > AV> You can include StoredFieldSortFactory class source file into your
> > > sources and
> > > AV> then use StoredFieldSortFactory.create(sortFieldName,
> > > sortDescending) to get
> > > AV> Sort object for sorting query.
> > > AV> StoredFieldSortFactory source file can be extracted from
> > > LUCENE-769 patch or
> > > AV> from sharehound sources: http://sharehound.cvs.sourceforge.net/*checkout*/sharehound/jNetCrawler/src/java/org/apache/lucene/search/StoredFieldSortFactory.java
> > >
> > >
> > >
> > > What I am wondering about is that in the patch commetns
> > > (https://issues.apache.org/jira/browse/LUCENE-769 ) I see that there
> > > is
> > > written that patch solves the problem by using WeakHashMap, but
> > > actually
> > > in the downloaded StoredFieldSortFactory.java  file there is not used
> > > WeakHashMap. Another thing: In the comments in Lucene-769 issue there
> > > is
> > > mentioned something about classes like: WeakDocumentsCache and
> > > DocCachingIndexReader but I did not found them in Lucene source code
> > > neither as classes in StoredFieldSortFactory.java. So my questions
> > > are:
> > > 1. Is it enought to include the file StoredFieldSortFactory.java in
> > > the
> > > source code or there are also other classes that I have to douwnload
> > > and
> > > include?
> > > 2. Have I to use this DocCachingIndexReader instead of Reader that I
> > > currently use in cases when I expect OOMException and will use this
> > > patch?
> > >
> > > Thanks to all once again :),
> > > Ivan
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message