Mailing-List: contact lucene-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Reply-To: <rengels@ix.netcom.com>
From: "Robert Engels" <rengels@ix.netcom.com>
To: "Lucene Developers List" <lucene-dev@jakarta.apache.org>
Subject: RE: strange lucene search behavior?
Date: Tue, 20 Jan 2004 11:48:18 -0600
Message-ID: <LMENLAOACIBLMOIILNNNEEECDMAA.rengels@ix.netcom.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
In-Reply-To: <400D6595.4080504@apache.org>
Importance: Normal

Doug,

I changed the search to use the HitCollector, and then indeed, it only
requests the TermDocs for a term once, so in this case, the LRU would not
help at all for a single query.

The TermDocs LRU should help a lot for repeated common searches, (which is
common with our application), especially when filtering using a date range,
because almost always the user is looking for 'recent' documents.

I can work the LRU into the code base - my imp does not use the
LinkedHashMap, but rather a HashMap and an associated LinkedList, so there
would not be any 1.4 issues. Since the index is never rewritten, and if the
cache is per Index, I do not need to worry about a stale cache, correct?

Robert

-----Original Message-----
From: Doug Cutting [mailto:cutting@apache.org]
Sent: Tuesday, January 20, 2004 11:30 AM
To: Lucene Developers List
Subject: Re: strange lucene search behavior?


Robert,

Some time back someone benchmarked adding an LRU cache to
TermInfosReader and were unable to see any significant overall speedup
in query processing.  If you find otherwise, please submit a patch.
Java 1.4's LinkedHashMap would make the implementation of such a cache
very simple, but, unfortunately, not all Lucene users are using 1.4 yet.

Also, if you wish to retrieve all of the hits, rather than just a
portion, please use the HitCollector API rather than the Hits API.  The
Hits API is optimized for applications which are only displaying a few
of the top hits.

Doug

Robert Engels wrote:
> In working with Lucene, I notice that when performing searches, it
retrieves
> the documents for the same term multiple times. I think this may be
because
> the Hits collection only stores a certain number of items, but would it
not
> be better to just increase the size of the Hits collection, rather than
> perform the extra, relatively very expensive, read of the term docs.
>
> The following is the trace output from Lucene performing 2 single term
> searches, and a multiple term search: (notice that in each case, the
> documents for a term are asked for twice).
>
> expression = +epson, query = +text:epson
> findTermInfo() text:epson, time = 0
> SearchTermDocs, seek() on text:epson
> SearchTermDocs, seek() on text:epson [cached]
> find, hits = 224, query time = 16, doc (150) time = 15, total time = 31
>
> expression = +printer, query = +text:printer
> findTermInfo() text:printer, time = 16
> SearchTermDocs, seek() on text:printer
> SearchTermDocs, seek() on text:printer [cached]
> find, hits = 5358, query time = 62, doc (150) time = 282, total time = 344
>
> expression = +epson +printer, query = +text:epson +text:printer
> SearchTermDocs, seek() on text:epson [cached]
> SearchTermDocs, seek() on text:printer [cached]
> SearchTermDocs, seek() on text:epson [cached]
> SearchTermDocs, seek() on text:printer [cached]
> find, hits = 175, query time = 15, doc (150) time = 47, total time = 62
>
> In order to limit the performance hit, or implementation caches the
returned
> docs within a query (the [cached] tag), but it seems the issue would be
> better addressed by the Lucene engine.
>
> Any thoughts on this?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org