lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson" <>
Subject Re: Out of memory exception for big indexes
Date Fri, 06 Apr 2007 14:30:39 GMT
I can only shed a little light on a couple of points, see below.

On 4/6/07, Ivan Vasilev <> wrote:
> Hi All,
> I have the following problem - we have OutOfMemoryException when
> seraching on the indexes that are of size 20 - 40 GB and contain 10 - 15
> million docs.
> When we make searches we perform query that match all the results but we
> DO NOT fetch all the results - we fetch 100 of them. We also make
> sorting by using the class Sort and we really need result to be sorted
> on a field that is randomly defined by the user.
> So my questions are:

The problem I suspect is the sorting. As I understand, Lucene
builds internal caches for sorting and I suspect that this is the root
of your problem. You can test this by trying your problem queries
without sorting.

How much memory are you giving the JVM?

1) Have Lucene some restrictions on index size on which it can perform
> searches?

No theoretical ones that I know of, but practical ones at times. As
you are finding.

2) Is there some approach to estimate beforehand the RAM that will use
> Lucene for sertain query? I mean on what exactly depends this memory
> usage - on index size, on docs stored in the index, on size of this
> docs...

I'd like to know this myself. Hint, hint, hint....

3) Is there some approach to controll the used RAM. For example when
> searching not to exceed 1GB of used memory?
> 4) Is there some spcial approach to proceeding with such big indexes (we
> expect in near future even 60 -80 GB indexes).
> Best Regards,
> Ivan
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message