lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nilesh Bansal" <nileshban...@gmail.com>
Subject Re: Out of memory exception for big indexes
Date Sun, 08 Apr 2007 05:03:06 GMT
This seems like a very useful patch. Our application searches over 50
million doc in a 40GB index. We only have simple conjunctive queries
on a single field. Currently, the command line search program that
prints top-10 results requires at least 200mb memory. Our web
application, that searches the same index crashes with OOM when there
are more than 10-12 concurrent requests (heap size set to 3GB). Will
this patch help in such a situation?

It seems that there are some issues with this patch and that was the
reason it is not yet in the main source tree. Can someone please
summerize what are the downsides of using such an approach. It will be
really good if Lucene had it in main source tree and a flag to turn ON
or OFF this feature.

Bublic, can you tell me what exactly I need to do if I want to use this patch?

thanks
Nilesh

On 4/6/07, Bublic Online <abublic@gmail.com> wrote:
> Hi Ivan, Chris and all!
>
> I'm that contributor of LUCENE-769 and I recommend it too :)
> OutOfMemory error was one of main reasons for me to make it.
>
> Regards,
> Artem Vasiliev
>
> On 4/6/07, Chris Hostetter <hossman_lucene@fucit.org> wrote:
> >
> >
> > : The problem I suspect is the sorting. As I understand, Lucene
> > : builds internal caches for sorting and I suspect that this is the root
> > : of your problem. You can test this by trying your problem queries
> > : without sorting.
> >
> > if Sorting really is the cause of your problems, you may want to try out
> > this patch...
> >
> > https://issues.apache.org/jira/browse/LUCENE-769
> >
> > ...it *may* be advantageous in situations where memory is your most
> > constrained resource, and you are willing to sacrifice speed for sorting
> > ... it looks promising to me, but there haven't been any convincing
> > usecases/benchmarks of people finding it beneficial (other then the
> > original contributor)
> >
> > if you do try it, please post your comments in the issue.
> >
> >
> >
> > -Hoss
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>


-- 
Nilesh Bansal.
http://queens.db.toronto.edu/~nilesh/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message