lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wettin <karl.wet...@gmail.com>
Subject Re: Memory consumed by IndexSearcher
Date Tue, 22 Sep 2009 17:58:31 GMT
Hi Mindaugas,

it is - as you sort of point out - the readers associated with your  
searcher that consumes the memory, and not so much the searcher it  
self. Thing that consume the most memory is probably field norms (8  
bits per field and document unless omitted) and flyweighted terms  
(String#interal), things you can't really do that much about.

But it is hard to say for sure when you've said so little about what  
your index really looks like. How many fields, do you use norms, how  
many documents do you have, how many unique terms are there, etc?

Another thing that can consume a lot of memory is sorting. Not sure if  
that data is bound to the searcher or not though.



     karl

21 sep 2009 kl. 16.32 skrev Mindaugas Žakšauskas:

> Hi,
>
> I was wondering what would be sensible amount of memory IndexSearcher
> can consume? In my application we do retain reference to it for
> quicker searches; however I have become a bit worried for it being a
> memory hog. We are using Lucene 2.4.0 on 8 CPU Linux SMP box; JVM is
> Sun's 1.6.0_14 64-Bit Server VM.
>
> I am asking because I have ended up with IndexSearcher having Retained
> size [1] of 145M. All of this memory is being eaten by
> IndexSearcher::reader::subReaders[]. The reader is MultiSegmentReader
> and all subReaders are SegmentReader. My memory dump showed subReaders
> array having size of 37 SegmentReaders, 2 to 5 M each. I can send
> YouKit screenshot if anyone's interested.
>
> All of that should be viewed in the light of index size on the disk,
> which is only 22M.
>
> I appreciate that all of this memory can be used for legitimate
> purposes; however is there a way to know when does it go over sensible
> limit? Can there be a "sensible" limit at all? Also, is it possible to
> set the physical boundary the IndexSearcher would never go over?
>
> Thanks in advance for all answers.
>
> Regards,
> Mindaugas
>
> [1] http://www.yourkit.com/docs/80/help/sizes.jsp
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message