Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of peterlkeegan@gmail.com
 designates 209.85.146.179 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:mime-version:content-type;
        b=HmQu81xCh4XNlWbHnn9TQ9zo9NBSdpkk/NnCkZCjcYz30287jXezHlkWFKbriY0/v1KAiCuc/mN0rAcrvSzFaaONo5lTuU5zRgk9/w7u0ynaHU+NZsT0SvuL854PY8IYETf3wPtKidFAqmpXcg774u0yV2vkO2HhNG0csvrePgo=
Message-ID: <e994873a0708131133r785d5d52i87a5b1aaba6cbac7@mail.gmail.com>
Date: Mon, 13 Aug 2007 14:33:37 -0400
From: "Peter Keegan" <peterlkeegan@gmail.com>
To: java-user <java-user@lucene.apache.org>
Subject: SpanQuery and database join
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_188998_12322085.1187030017151"

------=_Part_188998_12322085.1187030017151
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

I've been experimenting with using SpanQuery to perform what is essentially
a limited type of database 'join'. Each document in the index contains 1 or
more 'rows' of meta data from another 'table'. The meta data are simple
tokens representing a column name/value pair ( e.g. color$red or
location$123).  Each row is represented by a span with a maximum token
length equal to the maximum number of meta data columns. If a column has
multiple values, they are all indexed at the same position ( e.g. color$red,
color$blue). All rows are added to a single field. The spans are 'separated'
from each other by introducing a position gap between them via '
Analyzer.getPositionIncrementGap'. This gap should be greater than the
number of columns in each span.

At query time, a SpanNearQuery is constructed to represent the meta data to
join. The 'slop' value is set to the maximum number of meta data columns
(minus 1). Using a simple Antlr parser, boolean span queries with AND, OR,
NOT can be constructed fairly easily. The SpanQuery is And'd to the main
query to build the final query.

This approach is flexible and pretty efficient because no stored fields or
external data are accessed at query time. Span queries are more expensive
compared than other queries, though. We measure performance via throughput
(as opposed to the response time for a single query), and the addition of a
SpanQuery reduced throughput by 5X for ordered spans and 10X for unordered
spans. Still, this may be acceptable for some applications, especially if
spans are not used on every query.

I thought this might interest some of you.

Peter

------=_Part_188998_12322085.1187030017151--