Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=mAyaSV2CS3DlvPf9cNF+X4GFWxKijui8iE1JEqHPmc+Xc80wVubI6NnyuCM66W9wSOodp9lppHLYnYrwt+rMKy2Df+IfV4DV4UoK3vJ3yhA5Kv0G03qsH1aQc3erkxdUL4VCtZsGnDJbRT3f1bfk2IltrISIRjtHsgG9nBBqn5s=
  ;
Message-ID: <20070406171036.2674.qmail@web53511.mail.re2.yahoo.com>
Date: Fri, 6 Apr 2007 10:10:36 -0700 (PDT)
From: Craig W Conway <craigwconway@yahoo.com>
Subject: Re: Out of memory exception for big indexes
To: java-user@lucene.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-1508323934-1175879436=:999"

--0-1508323934-1175879436=:999
Content-Type: text/plain; charset=ascii

Would it be fair to say that you can expect OutOfMemory errors if you run complex queries? ie sorts, boosts, weights...

My query looks like this:

+(pathNodeId_2976569:1^5.0 pathNodeId_2976969:1 pathNodeId_2976255:1 pathNodeId_2976571:1) +(pathClassId:1 pathClassId:346 pathClassId:314) -id:369


My OutOfMemory error occurs like so:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid4512.hprof ...
Heap dump file created [71421503 bytes in 2.640 secs]
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.index.MultiReader.norms(MultiReader.java:173)
        at org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:69)
        at org.apache.lucene.search.BooleanQuery$BooleanWeight2.scorer(BooleanQuery.java:355)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:130)
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:100)
        at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:66)
        at org.apache.lucene.search.Hits.<init>(Hits.java:45)
        at org.apache.lucene.search.Searcher.search(Searcher.java:45)
        at org.apache.lucene.search.Searcher.search(Searcher.java:37)

References:

http://www.opensubscriber.com/message/java-user@lucene.apache.org/1961376.html
http://www.opensubscriber.com/message/java-user@lucene.apache.org/6362024.html


----- Original Message ----
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
To: java-user@lucene.apache.org
Sent: Friday, April 6, 2007 8:20:21 AM
Subject: Re: Out of memory exception for big indexes

Ivane,

Sorts will eat your memory, and how much they use depends on what you store in them - ints, String, floats...
A profiler like JProfiler will tell you what's going on, who's eating your memory.

Otis
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simpy -- http://www.simpy.com/  -  Tag  -  Search  -  Share

----- Original Message ----
From: Ivan Vasilev <ivasilev@sirma.bg>
To: java-user@lucene.apache.org
Sent: Friday, April 6, 2007 7:09:38 AM
Subject: Out of memory exception for big indexes

Hi All,

I have the following problem - we have OutOfMemoryException when 
seraching on the indexes that are of size 20 - 40 GB and contain 10 - 15 
million docs.
When we make searches we perform query that match all the results but we 
DO NOT fetch all the results - we fetch 100 of them. We also make 
sorting by using the class Sort and we really need result to be sorted 
on a field that is randomly defined by the user.
So my questions are:
1) Have Lucene some restrictions on index size on which it can perform 
searches?
2) Is there some approach to estimate beforehand the RAM that will use 
Lucene for sertain query? I mean on what exactly depends this memory 
usage - on index size, on docs stored in the index, on size of this docs...
3) Is there some approach to controll the used RAM. For example when 
searching not to exceed 1GB of used memory?
4) Is there some spcial approach to proceeding with such big indexes (we 
expect in near future even 60 -80 GB indexes).


Best Regards,
Ivan

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


____________________________________________________________________________________
Bored stiff? Loosen up... 
Download and play hundreds of games for free on Yahoo! Games.
http://games.yahoo.com/games/front
--0-1508323934-1175879436=:999--