lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chuck Williams" <ch...@manawiz.com>
Subject RE: How to proceed with Bug 31841 - MultiSearcher problems with Similarity.docFreq() ?
Date Wed, 12 Jan 2005 00:01:00 GMT
Doug Cutting wrote:
  > I'm not sure exactly what you mean by "distribute the idf
information
  > out to the RemoteSearchable".  I think one might profitably
implement a
  > docFreq() cache in RemoteSearchable.  This could be a simple cache,
or
  > it could be fairly agressive, pre-fetching all the docFreqs.  (As an
  > optimization, it could only pre-fetch those greater than 1, and,
when a
  > term is not in the cache, assume its docFreq is 1.  As a lossy
  > optimization, it could only pre-fetch those greater than N, and
somehow
  > estimate those not in the cache.)  Is that what you meant?

I was thinking of the aggressive version with an index-time solution,
although I don't know the Lucene architecture for distributed indexing
and searching well enough to formulate the idea precisely.
Conceptually, I'd like each server that owns a slice of the index in a
distributed environment to have the complete docFreq data, i.e. to have
docFreq's that represent the collection as a whole, not just its index
slice.  If this was achieved at index-time, then the current
implementation would work at query time.  I.e., MultiSearch could send
the queries out to the remote Searcher's and these Searcher's could
consult their local indexes for the correct docFreq's to use.

Chuck

  > -----Original Message-----
  > From: Doug Cutting [mailto:cutting@apache.org]
  > Sent: Tuesday, January 11, 2005 3:46 PM
  > To: Lucene Developers List
  > Subject: Re: How to proceed with Bug 31841 - MultiSearcher problems
with
  > Similarity.docFreq() ?
  > 
  > Chuck Williams wrote:
  > > This is a nice solution!  By having MultiSearcher create the
Weight,
  > it
  > > can pass itself in as the searcher, thereby allowing the correct
  > > docFreq() method to be called.
  > 
  > Glad to hear it at least makes sense... Now I hope it works!
  > 
  > > I'm still left wondering if having MultiSearcher query all the
  > > RemoteSearchable's on every call to docFreq() within each
TermQuery,
  > > PhraseQuery, SpanQuery and PhrasePrefixQuery is the way to go long
  > term,
  > > although it seems like the best thing to do right now.  The calls
only
  > > happen when the Weight's are created, so maybe it's not too bad.
  > Longer
  > > term, it might be better to distribute the idf information out to
the
  > > RemoteSearchable's to minimize the required number of remote
accesses
  > > for each Query.
  > 
  > I'm not sure exactly what you mean by "distribute the idf
information
  > out to the RemoteSearchable".  I think one might profitably
implement a
  > docFreq() cache in RemoteSearchable.  This could be a simple cache,
or
  > it could be fairly agressive, pre-fetching all the docFreqs.  (As an
  > optimization, it could only pre-fetch those greater than 1, and,
when a
  > term is not in the cache, assume its docFreq is 1.  As a lossy
  > optimization, it could only pre-fetch those greater than N, and
somehow
  > estimate those not in the cache.)  Is that what you meant?
  > 
  > Doug
  > 
  >
---------------------------------------------------------------------
  > To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
  > For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


Mime
View raw message