lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Dunn <james_h_d...@yahoo.com>
Subject Re: Memory usage
Date Thu, 27 May 2004 16:21:55 GMT
Otis,

My app does run within Tomcat.  But when I started
getting these OutOfMemoryErrors I wrote a little unit
test to watch the memory usage without Tomcat in the
middle and I still see the memory usage.

Thanks,

Jim
--- Otis Gospodnetic <otis_gospodnetic@yahoo.com>
wrote:
> Sorry if I'm stating the obvious.  Is this happening
> in some
> stand-alone unit tests, or are you running things
> from some application
> and in some environment, like Tomcat, Jetty or in
> some non-web app?
> 
> Your queries are pretty big (although I recall some
> people using even
> bigger ones... but it all depends on the hardware
> they had), but are
> you sure running out of memory is due to Lucene, or
> could it be a leak
> in the app from which you are running queries?
> 
> Otis
> 
> 
> --- James Dunn <james_h_dunn@yahoo.com> wrote:
> > Doug,
> > 
> > We only search on analyzed text fields.  There are
> a
> > couple of additional fields in the index like
> > OBJECT_ID that are keywords but we don't search
> > against those, we only use them once we get a
> result
> > back to find the thing that document represents.
> > 
> > Thanks,
> > 
> > Jim
> > 
> > --- Doug Cutting <cutting@apache.org> wrote:
> > > It is cached by the IndexReader and lives until
> the
> > > index reader is 
> > > garbage collected.  50-70 searchable fields is a
> > > *lot*.  How many are 
> > > analyzed text, and how many are simply keywords?
> > > 
> > > Doug
> > > 
> > > James Dunn wrote:
> > > > Doug,
> > > > 
> > > > Thanks!  
> > > > 
> > > > I just asked a question regarding how to
> calculate
> > > the
> > > > memory requirements for a search.  Does this
> > > memory
> > > > only get used only during the search operation
> > > itself,
> > > > or is it referenced by the Hits object or
> anything
> > > > else after the actual search completes?
> > > > 
> > > > Thanks again,
> > > > 
> > > > Jim
> > > > 
> > > > 
> > > > --- Doug Cutting <cutting@apache.org> wrote:
> > > > 
> > > >>James Dunn wrote:
> > > >>
> > > >>>Also I search across about 50 fields but I
> don't
> > > >>
> > > >>use
> > > >>
> > > >>>wildcard or range queries. 
> > > >>
> > > >>Lucene uses one byte of RAM per document per
> > > >>searched field, to hold the 
> > > >>normalization values.  So if you search a 10M
> > > >>document collection with 
> > > >>50 fields, then you'll end up using 500MB of
> RAM.
> > > >>
> > > >>If you're using unanalyzed fields, then an
> easy
> > > >>workaround to reduce the 
> > > >>number of fields is to combine many in a
> single
> > > >>field.  So, instead of, 
> > > >>e.g., using an "f1" field with value "abc",
> and an
> > > >>"f2" field with value 
> > > >>"efg", use a single field named "f" with
> values
> > > >>"1_abc" and "2_efg".
> > > >>
> > > >>We could optimize this in Lucene.  If no
> values of
> > > >>an indexed field are 
> > > >>analyzed, then we could store no norms for the
> > > field
> > > >>and hence read none 
> > > >>into memory.  This wouldn't be too hard to
> > > >>implement...
> > > >>
> > > >>Doug
> > > >>
> > > >>
> > > > 
> > > >
> > >
> >
>
---------------------------------------------------------------------
> > > > 
> > > >>To unsubscribe, e-mail:
> > > >>lucene-user-unsubscribe@jakarta.apache.org
> > > >>For additional commands, e-mail:
> > > >>lucene-user-help@jakarta.apache.org
> > > >>
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 	
> > > > 		
> > > > __________________________________
> > > > Do you Yahoo!?
> > > > Friends.  Fun.  Try the all-new Yahoo!
> Messenger.
> > > > http://messenger.yahoo.com/ 
> > > > 
> > > >
> > >
> >
>
---------------------------------------------------------------------
> > > > To unsubscribe, e-mail:
> > > lucene-user-unsubscribe@jakarta.apache.org
> > > > For additional commands, e-mail:
> > > lucene-user-help@jakarta.apache.org
> > > > 
> > > 
> > >
> >
>
---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > > lucene-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail:
> > > lucene-user-help@jakarta.apache.org
> > > 
> > 
> > 
> > 
> > 	
> > 		
> > __________________________________
> > Do you Yahoo!?
> > Friends.  Fun.  Try the all-new Yahoo! Messenger.
> > http://messenger.yahoo.com/ 
> > 
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> > 
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> 



	
		
__________________________________
Do you Yahoo!?
Friends.  Fun.  Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/ 

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message