lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: Memory usage
Date Thu, 27 May 2004 12:53:40 GMT
Sorry if I'm stating the obvious.  Is this happening in some
stand-alone unit tests, or are you running things from some application
and in some environment, like Tomcat, Jetty or in some non-web app?

Your queries are pretty big (although I recall some people using even
bigger ones... but it all depends on the hardware they had), but are
you sure running out of memory is due to Lucene, or could it be a leak
in the app from which you are running queries?

Otis


--- James Dunn <james_h_dunn@yahoo.com> wrote:
> Doug,
> 
> We only search on analyzed text fields.  There are a
> couple of additional fields in the index like
> OBJECT_ID that are keywords but we don't search
> against those, we only use them once we get a result
> back to find the thing that document represents.
> 
> Thanks,
> 
> Jim
> 
> --- Doug Cutting <cutting@apache.org> wrote:
> > It is cached by the IndexReader and lives until the
> > index reader is 
> > garbage collected.  50-70 searchable fields is a
> > *lot*.  How many are 
> > analyzed text, and how many are simply keywords?
> > 
> > Doug
> > 
> > James Dunn wrote:
> > > Doug,
> > > 
> > > Thanks!  
> > > 
> > > I just asked a question regarding how to calculate
> > the
> > > memory requirements for a search.  Does this
> > memory
> > > only get used only during the search operation
> > itself,
> > > or is it referenced by the Hits object or anything
> > > else after the actual search completes?
> > > 
> > > Thanks again,
> > > 
> > > Jim
> > > 
> > > 
> > > --- Doug Cutting <cutting@apache.org> wrote:
> > > 
> > >>James Dunn wrote:
> > >>
> > >>>Also I search across about 50 fields but I don't
> > >>
> > >>use
> > >>
> > >>>wildcard or range queries. 
> > >>
> > >>Lucene uses one byte of RAM per document per
> > >>searched field, to hold the 
> > >>normalization values.  So if you search a 10M
> > >>document collection with 
> > >>50 fields, then you'll end up using 500MB of RAM.
> > >>
> > >>If you're using unanalyzed fields, then an easy
> > >>workaround to reduce the 
> > >>number of fields is to combine many in a single
> > >>field.  So, instead of, 
> > >>e.g., using an "f1" field with value "abc", and an
> > >>"f2" field with value 
> > >>"efg", use a single field named "f" with values
> > >>"1_abc" and "2_efg".
> > >>
> > >>We could optimize this in Lucene.  If no values of
> > >>an indexed field are 
> > >>analyzed, then we could store no norms for the
> > field
> > >>and hence read none 
> > >>into memory.  This wouldn't be too hard to
> > >>implement...
> > >>
> > >>Doug
> > >>
> > >>
> > > 
> > >
> >
> ---------------------------------------------------------------------
> > > 
> > >>To unsubscribe, e-mail:
> > >>lucene-user-unsubscribe@jakarta.apache.org
> > >>For additional commands, e-mail:
> > >>lucene-user-help@jakarta.apache.org
> > >>
> > > 
> > > 
> > > 
> > > 
> > > 	
> > > 		
> > > __________________________________
> > > Do you Yahoo!?
> > > Friends.  Fun.  Try the all-new Yahoo! Messenger.
> > > http://messenger.yahoo.com/ 
> > > 
> > >
> >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > lucene-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail:
> > lucene-user-help@jakarta.apache.org
> > > 
> > 
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> > lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > lucene-user-help@jakarta.apache.org
> > 
> 
> 
> 
> 	
> 		
> __________________________________
> Do you Yahoo!?
> Friends.  Fun.  Try the all-new Yahoo! Messenger.
> http://messenger.yahoo.com/ 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message