Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Subject: Re: Facet queries blow out the filterCache
To: solr-user@lucene.apache.org
References: <D2330173.6F84C%jwartes@whitepages.com>
From: Charlie Hull <charlie@flax.co.uk>
Message-ID: <560E3F03.6000206@flax.co.uk>
Date: Fri, 2 Oct 2015 09:23:31 +0100
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <D2330173.6F84C%jwartes@whitepages.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

On 01/10/2015 23:31, Jeff Wartes wrote:
> It still inserts if I address the core directly and use distrib=false.
>
> I’ve got a few collections sharing the same config, so it’s surprisingly
> annoying to
> change solrconfig.xml right now, but it seemed pretty clear the query is
> the thing being cached, since
> the cache size only changes when the query does.

Hi Jeff,

I think you may be hitting the same issue we found:
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201409.mbox/%3CCAGe-MLJ+6y1at+OunK3sGaCFF6zGtJq_Nin9_3SHN0kFuqXkBA@mail.gmail.com%3E

Distributed faceting uses the filter cache, where you wouldn't expect it 
to. The solution was to set facet.limit to -1.

Best

Charlie
>
>
>
> On 10/1/15, 3:01 PM, "Mikhail Khludnev" <mkhludnev@griddynamics.com> wrote:
>
>> hm..
>> This option was useful for introspecting cache content
>> https://wiki.apache.org/solr/SolrCaching#showItems It might help you to
>> find-out a cause.
>> I'm still blaming distributed requests, it expained here
>> https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-Over-Re
>> questParameters
>> eg does it happen if you run with distrib=false?
>>
>> On Fri, Oct 2, 2015 at 12:27 AM, Jeff Wartes <jwartes@whitepages.com>
>> wrote:
>>
>>>
>>> No change, still shows an insert per-request. As does a simplified
>>> request
>>> with only the facet params
>>> "&facet.field=city&facet=true"
>>>
>> by default it's 100
>> https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-Theface
>> t.limitParameter
>> and can cause filtering by values, it can be seen in logs, btw.
>>
>>>
>>> It’s definitely facet related though, facet=false eliminates the insert.
>>>
>>>
>>>
>>> On 10/1/15, 1:50 PM, "Mikhail Khludnev" <mkhludnev@griddynamics.com>
>>> wrote:
>>>
>>>> what if you set f.city.facet.limit=-1 ?
>>>>
>>>> On Thu, Oct 1, 2015 at 7:43 PM, Jeff Wartes <jwartes@whitepages.com>
>>>> wrote:
>>>>
>>>>>
>>>>> I’m doing some fairly simple facet queries in a two-shard 5.3
>>> SolrCloud
>>>>> index on fields like this:
>>>>>
>>>>> <field name="city" type="string" indexed="true" stored="false"
>>>>> docValues="true”/>
>>>>>
>>>>> that look something like this:
>>>>>
>>>
>>>>> q=...&fl=id,score&facet.field=city&facet=true&f.city.facet.mincount=1&f
>>>>> .c
>>>>> it
>>>>> y.facet.limit=50&rows=0&start=0&facet.method=fc
>>>>>
>>>>> (no, NOT facet.method=enum - the usage of the filterCache there is
>>>>> pretty
>>>>> well documented)
>>>>>
>>>>> Watching the filterCache stats, it appears that every one of these
>>>>> queries
>>>>> causes the "inserts" counter to be incremented by one. Distinct "q="
>>>>> queries also increase the "size", and eviction happens as normal. If
>>> I
>>>>> repeat the same query a few times, "lookups" is not incremented, so
>>>>> these
>>>>> entries generally appear to be completely wasted. (Although when
>>>>> running a
>>>>> lot of these queries, it appears as though a very small set also
>>>>> increment
>>>>> the "lookups" counter, but only a small set, and I haven’t figured
>>> out
>>>>> why
>>>>> some are special.)
>>>>>
>>>>> So the question is, why does this facet query have anything to do
>>> with
>>>>> the
>>>>> filterCache? This causes a huge amount of filterCache churn with no
>>>>> apparent benefit.
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Sincerely yours
>>>> Mikhail Khludnev
>>>> Principal Engineer,
>>>> Grid Dynamics
>>>>
>>>> <http://www.griddynamics.com>
>>>> <mkhludnev@griddynamics.com>
>>>
>>>
>>
>>
>> --
>> Sincerely yours
>> Mikhail Khludnev
>> Principal Engineer,
>> Grid Dynamics
>>
>> <http://www.griddynamics.com>
>> <mkhludnev@griddynamics.com>
>


-- 
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk