lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Dunn <james_h_d...@yahoo.com>
Subject Re: Memory usage
Date Wed, 26 May 2004 23:00:38 GMT
Doug,

We only search on analyzed text fields.  There are a
couple of additional fields in the index like
OBJECT_ID that are keywords but we don't search
against those, we only use them once we get a result
back to find the thing that document represents.

Thanks,

Jim

--- Doug Cutting <cutting@apache.org> wrote:
> It is cached by the IndexReader and lives until the
> index reader is 
> garbage collected.  50-70 searchable fields is a
> *lot*.  How many are 
> analyzed text, and how many are simply keywords?
> 
> Doug
> 
> James Dunn wrote:
> > Doug,
> > 
> > Thanks!  
> > 
> > I just asked a question regarding how to calculate
> the
> > memory requirements for a search.  Does this
> memory
> > only get used only during the search operation
> itself,
> > or is it referenced by the Hits object or anything
> > else after the actual search completes?
> > 
> > Thanks again,
> > 
> > Jim
> > 
> > 
> > --- Doug Cutting <cutting@apache.org> wrote:
> > 
> >>James Dunn wrote:
> >>
> >>>Also I search across about 50 fields but I don't
> >>
> >>use
> >>
> >>>wildcard or range queries. 
> >>
> >>Lucene uses one byte of RAM per document per
> >>searched field, to hold the 
> >>normalization values.  So if you search a 10M
> >>document collection with 
> >>50 fields, then you'll end up using 500MB of RAM.
> >>
> >>If you're using unanalyzed fields, then an easy
> >>workaround to reduce the 
> >>number of fields is to combine many in a single
> >>field.  So, instead of, 
> >>e.g., using an "f1" field with value "abc", and an
> >>"f2" field with value 
> >>"efg", use a single field named "f" with values
> >>"1_abc" and "2_efg".
> >>
> >>We could optimize this in Lucene.  If no values of
> >>an indexed field are 
> >>analyzed, then we could store no norms for the
> field
> >>and hence read none 
> >>into memory.  This wouldn't be too hard to
> >>implement...
> >>
> >>Doug
> >>
> >>
> > 
> >
>
---------------------------------------------------------------------
> > 
> >>To unsubscribe, e-mail:
> >>lucene-user-unsubscribe@jakarta.apache.org
> >>For additional commands, e-mail:
> >>lucene-user-help@jakarta.apache.org
> >>
> > 
> > 
> > 
> > 
> > 	
> > 		
> > __________________________________
> > Do you Yahoo!?
> > Friends.  Fun.  Try the all-new Yahoo! Messenger.
> > http://messenger.yahoo.com/ 
> > 
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> > 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> 



	
		
__________________________________
Do you Yahoo!?
Friends.  Fun.  Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/ 

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message