Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of frederik.kraus@gmail.com
 designates 209.85.161.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=date:from:to:message-id:subject:x-mailer:mime-version:content-type
         :content-transfer-encoding;
        b=PXnqWeIpcmDzsXNRsXjs+hawz/LuqRbJN2HrfLjWQzvyJwFz6WxHCSvllxnJ//4OM3
         8crEcwuTf/lzvyPD2BHDZgLDssVLnWchiRjkHxu27JE17oTUPK7UBtwb4UXXRANNTrJ1
         TKBbHI8BLnE6uFl0KdKBX90RCZbjGVJgK0pwA=
Date: Fri, 25 Feb 2011 21:53:16 +0100
From: Frederik Kraus <frederik.kraus@gmail.com>
To: dev@lucene.apache.org
Message-ID: <3A19624083334BE18BCC2B34C27F0202@gmail.com>
Subject: FilterQuery Performance Optimizations
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="4d6816bc_7ab49daf_6e46"
Content-Transfer-Encoding: 8bit

--4d6816bc_7ab49daf_6e46
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 8bit
Content-Disposition: inline

Hi Guys, 

testing performance of complex filter queries on a rather large index, I ran into a few points I'd like to share and put up for discussion:

Let's say we have the following two filter queries:

fq=someField:(123 OR 234 OR 235)
fq=someField:(234 OR 123 OR 235)

Currently the filterCache treats those two queries as two distinct queries, where really they are logically the same.

Wouldn't it make more sense to internally sort this kind of logical OR query to reduce the number of distinct queries and at the same time increase the cache hits?

This also applies to the "AND" case (multivalue), even though you can obviously circumvent that issue via splitting:

fq=someField:(234 AND 123 AND 235)

into:

fq=someField:234&fq=someField:123&fq=someField:235

Going even one step further - might it not make sense to split up OR queries into individual filterQueries (much like the AND case, but internally), and then creating a UNION instead of an intersection as with the standard fq-chaining


Fred.


--4d6816bc_7ab49daf_6e46
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

<div>
            <div>
                <span>Hi Guys,
                </span><span></span></div><div><span><br></span></div><di=
v>testing performance of complex filter queries on a rather large index, =
I ran into a few points I'd like to share and put up for discussion:</div=
><div><br></div><div>Let's say we have the following two filter queries:<=
/div><div><br></div><div>fq=3Dsome=46ield:(123 OR 234&nbsp;OR&nbsp;235)</=
div><div>fq=3Dsome=46ield:(234&nbsp;OR&nbsp;123&nbsp;OR&nbsp;235)</div><d=
iv><br></div><div>Currently the filterCache treats those two queries as t=
wo distinct queries, where really they are logically the same.</div><div>=
<br></div><div>Wouldn't it make more sense to internally sort this kind o=
f logical OR query to reduce the number of distinct queries and at the sa=
me time increase the cache hits=3F</div><div><br></div><div>This also app=
lies to the =22AND=22 case (multivalue), even though you can obviously ci=
rcumvent that issue via splitting:</div><div><br></div><div>fq=3Dsome=46i=
eld:(234&nbsp;AND 123&nbsp;AND 235)</div><div><br></div><div>into:</div><=
div><br></div><div>fq=3Dsome=46ield:234&amp;fq=3Dsome=46ield:123&amp;fq=3D=
some=46ield:235</div><div><br></div><div>Going even one step further - mi=
ght it not make sense to split up OR queries into individual filterQuerie=
s (much like the AND case, but internally), and then creating a UNION ins=
tead of an intersection as with the standard fq-chaining</div><div><br></=
div><div><br></div><div>=46red.</div><div><br></div><div><br></div><div><=
br></div>
        </div>
--4d6816bc_7ab49daf_6e46--