Mailing-List: contact lucene-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Users List" <lucene-user@jakarta.apache.org>
From: Ype Kingma <ykingma@xs4all.nl>
To: lucene-user@jakarta.apache.org
Subject: Re: Mixing database and lucene searches
Date: Tue, 11 May 2004 20:01:12 +0200
User-Agent: KMail/1.5.4
References: <s0a0b85b.001@n6mcgw16.cchmc.org>
In-Reply-To: <s0a0b85b.001@n6mcgw16.cchmc.org>
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200405112001.13090.ykingma@xs4all.nl>

On Tuesday 11 May 2004 17:26, Gerard Sychay wrote:
> >>> "Eric Jain" <Eric.Jain@isb-sib.ch> 05/11/04 04:47AM >>>
> >>>
> > > Hits hits = searcher.search(new TermQuery("text", "foo")
> > > Set hitPKs = new Set();
> > > for each doc in hits:
> > >    hitPKs.put(doc.getField("pk"))
>
> > Retrieving even one custom field for every document of a possibly
> large
> > data set
> > can end up being very slow, it seems. This complicates things a
>
> lot...
>
> Glen, I don't know your application specifics, but if you are paging
> results, there is no need to retrieve all the primary keys at once.  I
> had a similar problem.  I ended up doing the following:
>
> - Store ONLY the primary keys in the index.  Ideally, you only  need
> two fields per Lucene Document: the tokenized text to be searched, and
> stored corresponding primary key.
> - Upon searching, get all the Hits as normal, say you get 10000 hits.
> - But the first page only displays first 10 hits, so retrieve first 10
> primary keys from the Hits, use these to form a SQL query and retrieve
> any info you need from the DB.  This way, you only handling 10 documents
> at a time, or howevery many per page.
> In actual use, this is very fast.

A Hits object caches some documents for you, but when you need more
control over stored field retrieval you can implement your own
to retrieve them in the way a database does it:
Get all the doc nrs needed, sort them and retrieve the (non cached) stored
fields in that order. Normally, that (almost) minimizes the distance the disk head
needs to travel for the retrieval.

Lucene stores all the document fields for single document close
together, so retrieving all stored fields isn't much more expensive than
retrieving only the primary key.

Kind regards,
Ype


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org