lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bob Daha <neoka...@yahoo.com>
Subject Re: Out of memory?
Date Mon, 10 Dec 2007 22:43:16 GMT
Thanks for the reply.  The backup of the index on disc is around 80 MB.  I'm giving my JVM
4 gigs of heap space...  I probably should have mentioned in my first thread that this is
not the first crash; it's done it a few times now and usually has almost a week of uptime
before it crashes even with the growing index.  (last time it crashed the index was about
60MB on disc)  It's also worth noting I am only going to keep around 4 weeks of ticket data
and I'm adding week 5 right now, so the index should be relatively close to this size "forever",
although rather rapidly changing.

I think *something* is holding onto those terms... just not sure what.  =\



----- Original Message ----
From: Grant Ingersoll <gsingers@apache.org>
To: java-user@lucene.apache.org
Sent: Monday, December 10, 2007 2:24:58 PM
Subject: Re: Out of memory?


Those terms could be coming from a lot of diff. places.

What size JVM heap, etc. are you giving your application?  My guess is
  
your index is too big to fit into RAM, you probably need to use a  
FSDirectory or give it more RAM, but it is hard to say for sure w/o  
knowing more about the application.

See http://wiki.apache.org/lucene-java/BasicsOfPerformance for more on
  
performance.

-Grant

On Dec 10, 2007, at 2:56 PM, Bob Daha wrote:

> Hello,
>
> I'm building a ticketing system for my company and am using Lucene  
> for some of the more complicated queries.  I'd say my application  
> differs from the typical lucene application in that my documents are
  
> (re)-indexed more frequently, the query load is actually relatively  
> light, and most of the indexed document data is numerical.  (ticket  
> id, owner id, date ranges, etc are numbers and there's just one  
> giant "text" searchable field.)
>
> I just got an out of memory exception on my app.  When I fired up  
> jhat, I was very surprised at the object allocation:
>
> 7862 instances of class org.apache.lucene.index.Term
> 7787 instances of class org.apache.lucene.index.TermInfo
> 504 instances of class org.apache.lucene.index.FieldInfo
> 224 instances of class org.apache.lucene.index.TermBuffer
> 79 instances of class org.apache.lucene.index.CompoundFileReader 
> $CSIndexInput
> 75 instances of class org.apache.lucene.index.SegmentTermEnum
> 70 instances of class org.apache.lucene.store.RAMFile
> 64 instances of class org.apache.lucene.store.RAMInputStream
> 63 instances of class org.apache.lucene.index.FieldInfos
> 20 instances of class
 com.facebook.tps_search_server.tps_search_ticket
> 20 instances of class  
> com.facebook.tps_search_server.tps_search_ticket$Isset
> 10 instances of class org.apache.lucene.index.IndexReader$FieldOption
> 9 instances of class
 org.apache.lucene.index.IndexFileDeleter$RefCount
> 8 instances of class org.apache.lucene.index.CompoundFileReader 
> $FileEntry
> 8 instances of class org.apache.lucene.index.CompoundFileWriter 
> $FileEntry
> 8 instances of class org.apache.lucene.index.SegmentReader$Norm
> 7 instances of class org.apache.lucene.document.FieldSelectorResult
> 5 instances of class org.apache.lucene.document.Field$TermVector
> 5 instances of class org.apache.lucene.queryParser.Token
> 4 instances of class org.apache.lucene.document.Field$Index
> 4 instances of class org.apache.lucene.search.FieldCacheImpl$Entry
> 3 instances of class com.facebook.thrift.protocol.TBinaryProtocol
> 3 instances of class com.facebook.thrift.transport.TSocket
> 3 instances of class org.apache.lucene.document.Field$Store
> 3 instances of class org.apache.lucene.index.SegmentInfos
> 3 instances of class org.apache.lucene.search.BooleanClause$Occur
> 2 instances of class com.facebook.thrift.protocol.TBinaryProtocol 
> $Factory
> 2 instances of class com.facebook.thrift.server.TThreadPoolServer 
> $WorkerProcess
> 2 instances of class com.facebook.thrift.transport.TTransportFactory
> 2 instances of class  
> org.apache.lucene.analysis.standard.StandardAnalyzer
> 2 instances of class org.apache.lucene.index.SegmentInfo
> 2 instances of class org.apache.lucene.queryParser.QueryParser 
> $Operator
> 2 instances of class org.apache.lucene.search.DefaultSimilarity
> 2 instances of class org.apache.lucene.search.FieldSortedHitQueue$2
> 2 instances of class org.apache.lucene.search.Sort
> 2 instances of class org.apache.lucene.search.SortField
> 2 instances of class org.apache.lucene.store.RAMDirectory
> 2 instances of class
 org.apache.lucene.store.SingleInstanceLockFactory
> 2 instances of class [Lorg.apache.lucene.search.SortField;
> ...
>
> Any ideas on why I'd have so many Term and TermInfo objects?  To  
> give you a little more insight into my application, worker threads  
> can be spawned by client requests that do one of 4 things - search,  
> index tickets, deindex tickets, or reindex tickets.  (reindex a  
> ticket is basically deindex then index ticket)  I'm using a  
> RAMDirectory for all my searches and updates although I have another
  
> thread which backs this up to disc periodically.
>
> Thanks,
> B
>
>
>
>
>       
>
 ____________________________________________________________________________________
> Looking for last minute shopping deals?
> Find them fast with Yahoo! Search.
  http://tools.search.yahoo.com/newsearch/category.php?category=shopping

--------------------------
Grant Ingersoll
http://lucene.grantingersoll.com

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org







      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message