lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Lu" <chris...@gmail.com>
Subject Re: Out of memory?
Date Mon, 10 Dec 2007 22:52:30 GMT
It's hard to debug just by your description. But I think in general,
you should close the reader and searcher, after you update the index.

BTW: If you are using database, my software DBSight does this kind of
pruning out old data and keep the index up-to-date. No memory leaks
etc. It's not related to how you search, but how you release the
resources.

-- 
Chris Lu
-------------------------
Instant Scalable Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
Lucene Database Search in 3 minutes:
http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
DBSight customer Twenga got 2.6 Million Euro funding!
http://www.techcrunch.com/2007/12/07/twenga-gets-e26-million-for-product-search/



On Dec 10, 2007 2:43 PM, Bob Daha <neokalus@yahoo.com> wrote:
> Thanks for the reply.  The backup of the index on disc is around 80 MB.  I'm giving my
JVM 4 gigs of heap space...  I probably should have mentioned in my first thread that this
is not the first crash; it's done it a few times now and usually has almost a week of uptime
before it crashes even with the growing index.  (last time it crashed the index was about
60MB on disc)  It's also worth noting I am only going to keep around 4 weeks of ticket data
and I'm adding week 5 right now, so the index should be relatively close to this size "forever",
although rather rapidly changing.
>
> I think *something* is holding onto those terms... just not sure what.  =\
>
>
>
> ----- Original Message ----
> From: Grant Ingersoll <gsingers@apache.org>
> To: java-user@lucene.apache.org
> Sent: Monday, December 10, 2007 2:24:58 PM
> Subject: Re: Out of memory?
>
>
> Those terms could be coming from a lot of diff. places.
>
> What size JVM heap, etc. are you giving your application?  My guess is
>
> your index is too big to fit into RAM, you probably need to use a
> FSDirectory or give it more RAM, but it is hard to say for sure w/o
> knowing more about the application.
>
> See http://wiki.apache.org/lucene-java/BasicsOfPerformance for more on
>
> performance.
>
> -Grant
>
>
> On Dec 10, 2007, at 2:56 PM, Bob Daha wrote:
>
> > Hello,
> >
> > I'm building a ticketing system for my company and am using Lucene
> > for some of the more complicated queries.  I'd say my application
> > differs from the typical lucene application in that my documents are
>
> > (re)-indexed more frequently, the query load is actually relatively
> > light, and most of the indexed document data is numerical.  (ticket
> > id, owner id, date ranges, etc are numbers and there's just one
> > giant "text" searchable field.)
> >
> > I just got an out of memory exception on my app.  When I fired up
> > jhat, I was very surprised at the object allocation:
> >
> > 7862 instances of class org.apache.lucene.index.Term
> > 7787 instances of class org.apache.lucene.index.TermInfo
> > 504 instances of class org.apache.lucene.index.FieldInfo
> > 224 instances of class org.apache.lucene.index.TermBuffer
> > 79 instances of class org.apache.lucene.index.CompoundFileReader
> > $CSIndexInput
> > 75 instances of class org.apache.lucene.index.SegmentTermEnum
> > 70 instances of class org.apache.lucene.store.RAMFile
> > 64 instances of class org.apache.lucene.store.RAMInputStream
> > 63 instances of class org.apache.lucene.index.FieldInfos
> > 20 instances of class
>  com.facebook.tps_search_server.tps_search_ticket
> > 20 instances of class
> > com.facebook.tps_search_server.tps_search_ticket$Isset
> > 10 instances of class org.apache.lucene.index.IndexReader$FieldOption
> > 9 instances of class
>  org.apache.lucene.index.IndexFileDeleter$RefCount
> > 8 instances of class org.apache.lucene.index.CompoundFileReader
> > $FileEntry
> > 8 instances of class org.apache.lucene.index.CompoundFileWriter
> > $FileEntry
> > 8 instances of class org.apache.lucene.index.SegmentReader$Norm
> > 7 instances of class org.apache.lucene.document.FieldSelectorResult
> > 5 instances of class org.apache.lucene.document.Field$TermVector
> > 5 instances of class org.apache.lucene.queryParser.Token
> > 4 instances of class org.apache.lucene.document.Field$Index
> > 4 instances of class org.apache.lucene.search.FieldCacheImpl$Entry
> > 3 instances of class com.facebook.thrift.protocol.TBinaryProtocol
> > 3 instances of class com.facebook.thrift.transport.TSocket
> > 3 instances of class org.apache.lucene.document.Field$Store
> > 3 instances of class org.apache.lucene.index.SegmentInfos
> > 3 instances of class org.apache.lucene.search.BooleanClause$Occur
> > 2 instances of class com.facebook.thrift.protocol.TBinaryProtocol
> > $Factory
> > 2 instances of class com.facebook.thrift.server.TThreadPoolServer
> > $WorkerProcess
> > 2 instances of class com.facebook.thrift.transport.TTransportFactory
> > 2 instances of class
> > org.apache.lucene.analysis.standard.StandardAnalyzer
> > 2 instances of class org.apache.lucene.index.SegmentInfo
> > 2 instances of class org.apache.lucene.queryParser.QueryParser
> > $Operator
> > 2 instances of class org.apache.lucene.search.DefaultSimilarity
> > 2 instances of class org.apache.lucene.search.FieldSortedHitQueue$2
> > 2 instances of class org.apache.lucene.search.Sort
> > 2 instances of class org.apache.lucene.search.SortField
> > 2 instances of class org.apache.lucene.store.RAMDirectory
> > 2 instances of class
>  org.apache.lucene.store.SingleInstanceLockFactory
> > 2 instances of class [Lorg.apache.lucene.search.SortField;
> > ...
> >
> > Any ideas on why I'd have so many Term and TermInfo objects?  To
> > give you a little more insight into my application, worker threads
> > can be spawned by client requests that do one of 4 things - search,
> > index tickets, deindex tickets, or reindex tickets.  (reindex a
> > ticket is basically deindex then index ticket)  I'm using a
> > RAMDirectory for all my searches and updates although I have another
>
> > thread which backs this up to disc periodically.
> >
> > Thanks,
> > B
> >
> >
> >
> >
> >
> >
>  ____________________________________________________________________________________
> > Looking for last minute shopping deals?
> > Find them fast with Yahoo! Search.
>   http://tools.search.yahoo.com/newsearch/category.php?category=shopping
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
>
>
>       ____________________________________________________________________________________
> Never miss a thing.  Make Yahoo your home page.
> http://www.yahoo.com/r/hs

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message