lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Savvas-Andreas Moysidis <savvas.andreas.moysi...@googlemail.com>
Subject Re: Displaying search result data - stored fields vs external source
Date Thu, 17 Sep 2009 13:50:11 GMT
Hello ,

I would also prefer to store the content in the index because, as Erick
points out this leads to a more simplified design but also because it allows
me to preserve the relevance sort.



If you store only  the item id in the index then when extracting all the
other required data from  supposedly a database, you will probably execute a
“*select * from item where id in (id_1,id_2,id_3...)*”

which will probably not retain your relevance sort. So unless you sort by a
business field or apply some kind of convoluted sort strategy which maps to
your original Lucene ResultSet you will have lost your ranking.



Cheers,

savvas


2009/9/15 Erick Erickson <erickerickson@gmail.com>

> Categorically I store everything in the index unless/until I *know* it
> doesn'twork. With some things, it's easy to know from the outset, like if I
> have
> 20T of data to store.
>
> First, storing fields has minimal impact on the search speed, the stored
> text
> isn't interleaved with the search tokens, so they're pretty much disjoint.
>
> Second, any scheme storing data separately is inherently more complex
> and difficult to maintain. From the eXtreme Programming folks "Do the
> simplest thing that could possibly work".
>
> Third, there isn't much work in trying it and seeing. I mean you have to
> write
> the retrieval code, and if you encapsulate fetching the data you can switch
> it out later if it comes to that pretty easily. So you don't lose much at
> all
> by "just trying it" <G>..
>
> HTH
> Erick
>
> On Tue, Sep 15, 2009 at 4:19 AM, Joel Halbert <joel@su3analytics.com>
> wrote:
>
> > Hi,
> >
> > When using Lucene I always consider two approaches to displaying search
> > result data to users:
> >
> > 1. Store any fields that we index and display to users in the Lucene
> > Documents themselves. When we perform a search simply retrieve the data
> > to be displayed from the Lucence documents themselves.
> >
> > or
> >
> > 2. Index fields in Lucene but reference data to be displayed from
> > another source, such as a database. So, when searching I would search
> > for documents then use a (stored) reference key on the documents to then
> > lookup the display fields to display from another source e.g. a
> > database.
> >
> > With regards to the number and size of stored fields I am looking at
> > indexing and displaying approximately 4 relatively small fields for each
> > document (e.g.  name, age, short description, URL ~ approx 500bytes in
> > total). In any query about 10 hits will be displayed to the user.
> > Approximately 10 million documents to index and search.
> >
> > I am interested the differences in both approaches with regards to:
> >
> > 1) Indexing time performance (how long it might take to index with and
> > without stored fields)
> > 2) Search time performance (total time taken to search for matching
> > documents and then display fields to users)
> >
> > I am less interested in differences arising from
> > maintainability/increased storage requirements.
> >
> > I would be interested to see what others  think of using each approach.
> >
> > Cheers,
> > Joel
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message