Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of yseeley@gmail.com designates
 209.85.128.169 as permitted sender)
MIME-Version: 1.0
Sender: yseeley@gmail.com
In-Reply-To: 
 <CAC8vSzb9YSy7R5pHmY2pZDrHzO0J5k-b+=7Pe9_KMxr_0cxk1g@mail.gmail.com>
References: 
 <CAC8vSzb9YSy7R5pHmY2pZDrHzO0J5k-b+=7Pe9_KMxr_0cxk1g@mail.gmail.com>
Date: Tue, 3 Jun 2014 17:19:08 -0400
Message-ID: 
 <CAB_8Yd-AL3yGypWFFSgRDrJ6VrY4ZURjWy8ymbnnXBWKkSg-1w@mail.gmail.com>
Subject: Re: Is the act of *caching* an fq very expensive? (seems to cost 4
 seconds in my example)
From: Yonik Seeley <yonik@heliosearch.com>
To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On Tue, Jun 3, 2014 at 4:44 PM, Brett Hoerner <brett@bretthoerner.com> wrot=
e:
> If I run a query like this,
>
> fq=3Dtext:lol
> fq=3Dcreated_at_tdid:[1400544000 TO 1400630400]
>
> It takes about 6 seconds. Following queries take only 50ms or less, as
> expected because my fqs are cached.
>
> However, if I change the query to not cache my big range query:
>
> fq=3Dtext:lol
> fq=3D{!cache=3Dfalse}created_at_tdid:[1400544000 TO 1400630400]
>
> It takes 2 seconds every time, which is a much better experience for my
> "first query for that range."
>
> What's odd to me is that I would expect both of these (first) queries to
> have to do the same amount of work, expect the first one stuffs the
> resulting bitset into a map at the end... which seems to have a 4 second
> overhead?

They are not equivalent.  Caching the filter separately (so it can be
reused in any combination with other queries and filters) means that
*all* docs that match the filter are collected and cached (it's the
collection that is taking the time).

For the {!cache=3Dfalse} case, Solr executes different code (it doesn't
just skip the caching step).

http://heliosearch.org/advanced-filter-caching-in-solr/

"""When a filter isn=E2=80=99t generated up front and cached, it=E2=80=99s =
executed in
parallel with the main query. First, the filter is asked about the
first document id that it matches. The query is then asked about the
first document that is equal to or greater than that document. The
filter is then asked about the first document that is equal to or
greater than that. The filter and the query play this game of leapfrog
until they land on the same document and it=E2=80=99s declared a match, aft=
er
which the document is collected and scored."""

So try:
  q=3D*:*
  fq=3Dcreated_at_tdid:[1400544000 TO 1400630400]

-Yonik
http://heliosearch.org - facet functions, subfacets, off-heap filters&field=
cache