lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Dunn <james_h_d...@yahoo.com>
Subject Re: Memory usage
Date Wed, 26 May 2004 21:36:51 GMT
Erik,

Thanks for the response.  

My actual documents are fairly small.  Most docs only
have about 10 fields.  Some of those fields are
stored, however, like the OBJECT_ID, NAME and DESC
fields.  The stored fields are pretty small as well. 
None should be more than 4KB and very few will
approach that limit.

I'm also using the default maxFieldSize value of
10000.  

I'm not caching hits, either.

Could it be my query?  I have about 80 total unique
fields in the index although no document has all 80. 
My query ends up looking like this:

+(F1:test F2:test ......  F80:test)

>From previous mails that doesn't look like an enormous
amount of fields to be searching against.  Is there
some formula for the amount of memory required for a
query based on the number of clauses and terms?

Jim



--- Erik Hatcher <erik@ehatchersolutions.com> wrote:
> How big are your actual Documents?  Are you caching
> Hits?  It stores, 
> internally, up to 200 documents.
> 
> 	Erik
> 
> 
> On May 26, 2004, at 4:08 PM, James Dunn wrote:
> 
> > Will,
> >
> > Thanks for your response.  It may be an object
> leak.
> > I will look into that.
> >
> > I just ran some more tests and this time I create
> a
> > 20GB index by repeatedly merging my large index
> into
> > itself.
> >
> > When I ran my test query against that index I got
> an
> > OutOfMemoryError on the very first query.  I have
> my
> > heap set to 512MB.  Should a query against a 20GB
> > index require that much memory?  I page through
> the
> > results 100 at a time, so I should never have more
> > than 100 Document objects in memory.
> >
> > Any help would be appreciated, thanks!
> >
> > Jim
> > --- wallen@Cyveillance.com wrote:
> >> This sounds like a memory leakage situation.  If
> you
> >> are using tomcat I
> >> would suggest you make sure you are on a recent
> >> version, as it is known to
> >> have some memory leaks in version 4.  It doesn't
> >> make sense that repeated
> >> queries would use more memory that the most
> >> demanding query unless objects
> >> are not getting freed from memory.
> >>
> >> -Will
> >>
> >> -----Original Message-----
> >> From: James Dunn [mailto:james_h_dunn@yahoo.com]
> >> Sent: Wednesday, May 26, 2004 3:02 PM
> >> To: lucene-user@jakarta.apache.org
> >> Subject: Memory usage
> >>
> >>
> >> Hello,
> >>
> >> I was wondering if anyone has had problems with
> >> memory
> >> usage and MultiSearcher.
> >>
> >> My index is composed of two sub-indexes that I
> >> search
> >> with a MultiSearcher.  The total size of the
> index
> >> is
> >> about 3.7GB with the larger sub-index being 3.6GB
> >> and
> >> the smaller being 117MB.
> >>
> >> I am using Lucene 1.3 Final with the compound
> file
> >> format.
> >>
> >> Also I search across about 50 fields but I don't
> use
> >> wildcard or range queries.
> >>
> >> Doing repeated searches in this way seems to
> >> eventually chew up about 500MB of memory which
> seems
> >> excessive to me.
> >>
> >> Does anyone have any ideas where I could look to
> >> reduce the memory my queries consume?
> >>
> >> Thanks,
> >>
> >> Jim
> >>
> >>
> >> 	
> >> 		
> >> __________________________________
> >> Do you Yahoo!?
> >> Friends.  Fun.  Try the all-new Yahoo! Messenger.
> >> http://messenger.yahoo.com/
> >>
> >>
> >
>
---------------------------------------------------------------------
> >> To unsubscribe, e-mail:
> >> lucene-user-unsubscribe@jakarta.apache.org
> >> For additional commands, e-mail:
> >> lucene-user-help@jakarta.apache.org
> >>
> >>
> >
>
---------------------------------------------------------------------
> >> To unsubscribe, e-mail:
> >> lucene-user-unsubscribe@jakarta.apache.org
> >> For additional commands, e-mail:
> >> lucene-user-help@jakarta.apache.org
> >>
> >
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam
> protection around
> > http://mail.yahoo.com
> >
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> 



	
		
__________________________________
Do you Yahoo!?
Friends.  Fun.  Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/ 

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message