Mailing-List: contact lucene-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Lucene Users List" <lucene-user@jakarta.apache.org>
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: Fields
Date: Tue, 11 May 2004 18:39:11 -0700
Message-ID: <5238CD8601F3EF4BA3C5553FECBB7D2A027F7BAA@exchange.qsent.com>
Thread-Topic: Fields
Thread-Index: AcQ3vjyEu9k32q5JRcuohU2Ez3U2xgAANZig
From: "Ryan Aslett" <Ryan.Aslett@Qsent.com>
To: "Lucene Users List" <lucene-user@jakarta.apache.org>

How much of a performance benefit/impact does "fielding" your data have
in Lucene?

Lets say I have 100 million documents.  I have Name, Phone, and Address
for each document.

I could either index the terms in separate fields, like=20
Field.Text("Name","Bob Jones");
Field.Keyword("Phone","5551212");
Field.Text("Address","123 Main");

Or, I could make everything in the same field, prepending a field
designator to the term itself as keywords, like:
Field.Keyword("Universal","nmBob");
Field.Keyword("Universal","nmJones");
Field.Keyword("Universal","ph5551212");
Field.Keyword("Universal","ad123");
Field.Keyword("Universal","adMain");

And when I build my queries always seach the same field, and prepend the
"fieldcode" to the search term.

Lets also assume that these universal fields are only indexed and not
stored, and I store something completely different as the actual stored
data.

Assumptions:=20
*Indexing/Preprocessing speed isnt important, unless its orders of
magnitude slower.
*10 indexes of 10 million Documents each.

Does anybody have any ideas as to the impact on query performance with
this method? Pros/Cons?

A commercial product that we are using is much slower when "fielding"
data, and has the concept of "unfielded literals". This second method is
how we currently field data and it seems to give us a tremendous
performance boost. Im curious if Lucene works in a similar fashion...

Ryan Aslett

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org