Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of peterlkeegan@gmail.com
 designates 209.85.146.178 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=T3DcIWZfb2CO1Q1zm0QFkxhaN6Ujg2cjTTebpbE2pgB53UsThyWOCWdb/Sw+LBOu3id5XVtELqwICTyK0/n33FV4r9nuc6HQS74qQR/7m6d0pvq6mzFS3vJ/HqNBOJ8qYCb3IvRqz8PCbhvZVGtRrT/yuxBHcrxGCl5iH+aqQR8=
Message-ID: <e994873a0708131226w4f13b182jade375839da4d353@mail.gmail.com>
Date: Mon, 13 Aug 2007 15:26:58 -0400
From: "Peter Keegan" <peterlkeegan@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: SpanQuery and database join
In-Reply-To: <359a92830708131209p62e9988bw9416d9af0fcf2711@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_189971_26262805.1187033218638"
References: <e994873a0708131133r785d5d52i87a5b1aaba6cbac7@mail.gmail.com>
	 <359a92830708131209p62e9988bw9416d9af0fcf2711@mail.gmail.com>

------=_Part_189971_26262805.1187033218638
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

I suppose it could go under performance or HowTo/Interesting uses of
SpanQuery.

Peter

On 8/13/07, Erick Erickson <erickerickson@gmail.com> wrote:
>
> Thanks for writing this up. Do you think this is an appropriate subject
> for the Wiki performance page?
>
> Erick
>
> On 8/13/07, Peter Keegan <peterlkeegan@gmail.com> wrote:
> >
> > I've been experimenting with using SpanQuery to perform what is
> > essentially
> > a limited type of database 'join'. Each document in the index contains 1
> > or
> > more 'rows' of meta data from another 'table'. The meta data are simple
> > tokens representing a column name/value pair ( e.g. color$red or
> > location$123).  Each row is represented by a span with a maximum token
> > length equal to the maximum number of meta data columns. If a column has
> > multiple values, they are all indexed at the same position ( e.g.
> > color$red,
> > color$blue). All rows are added to a single field. The spans are
> > 'separated'
> > from each other by introducing a position gap between them via '
> > Analyzer.getPositionIncrementGap'. This gap should be greater than the
> > number of columns in each span.
> >
> > At query time, a SpanNearQuery is constructed to represent the meta data
> > to
> > join. The 'slop' value is set to the maximum number of meta data columns
> > (minus 1). Using a simple Antlr parser, boolean span queries with AND,
> OR,
> > NOT can be constructed fairly easily. The SpanQuery is And'd to the main
> > query to build the final query.
> >
> > This approach is flexible and pretty efficient because no stored fields
> or
> > external data are accessed at query time. Span queries are more
> expensive
> > compared than other queries, though. We measure performance via
> throughput
> > (as opposed to the response time for a single query), and the addition
> of
> > a
> > SpanQuery reduced throughput by 5X for ordered spans and 10X for
> unordered
> > spans. Still, this may be acceptable for some applications, especially
> if
> > spans are not used on every query.
> >
> > I thought this might interest some of you.
> >
> > Peter
> >
>

------=_Part_189971_26262805.1187033218638--