lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Erickson <erickerick...@gmail.com>
Subject Re: Displaying search result data - stored fields vs external source
Date Tue, 15 Sep 2009 13:55:38 GMT
Categorically I store everything in the index unless/until I *know* it
doesn'twork. With some things, it's easy to know from the outset, like if I
have
20T of data to store.

First, storing fields has minimal impact on the search speed, the stored
text
isn't interleaved with the search tokens, so they're pretty much disjoint.

Second, any scheme storing data separately is inherently more complex
and difficult to maintain. From the eXtreme Programming folks "Do the
simplest thing that could possibly work".

Third, there isn't much work in trying it and seeing. I mean you have to
write
the retrieval code, and if you encapsulate fetching the data you can switch
it out later if it comes to that pretty easily. So you don't lose much at
all
by "just trying it" <G>..

HTH
Erick

On Tue, Sep 15, 2009 at 4:19 AM, Joel Halbert <joel@su3analytics.com> wrote:

> Hi,
>
> When using Lucene I always consider two approaches to displaying search
> result data to users:
>
> 1. Store any fields that we index and display to users in the Lucene
> Documents themselves. When we perform a search simply retrieve the data
> to be displayed from the Lucence documents themselves.
>
> or
>
> 2. Index fields in Lucene but reference data to be displayed from
> another source, such as a database. So, when searching I would search
> for documents then use a (stored) reference key on the documents to then
> lookup the display fields to display from another source e.g. a
> database.
>
> With regards to the number and size of stored fields I am looking at
> indexing and displaying approximately 4 relatively small fields for each
> document (e.g.  name, age, short description, URL ~ approx 500bytes in
> total). In any query about 10 hits will be displayed to the user.
> Approximately 10 million documents to index and search.
>
> I am interested the differences in both approaches with regards to:
>
> 1) Indexing time performance (how long it might take to index with and
> without stored fields)
> 2) Search time performance (total time taken to search for matching
> documents and then display fields to users)
>
> I am less interested in differences arising from
> maintainability/increased storage requirements.
>
> I would be interested to see what others  think of using each approach.
>
> Cheers,
> Joel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message