lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: FilterCache issue
Date Thu, 18 Jun 2009 12:35:33 GMT
Thats why I asked about multi-valued terms. If hes not using the enum 
faceting method (which only makes sense with fewer uniques), and the 
fields are not multi-valued, than it is using the FieldCache method. 
Which of course does use the filterCache, and works best when the 
filterCache size is the size of all the unique terms in all the fields 
you are faceting on.

Perhaps his machine can't handle that, but certainly he would benefit 
heavily from the cache. And if you had tons of uniques and a small 
cache, you would see exactly what he is seeing. Probably best to see how 
it goes before seeing if you have to optimize based on garbage 
collection (and I beleive that he could run in to resource issues, whats 
why I said give it a shot :) ). He may just need more resources than he 
can get, but I'm fairly sure he needs the resources. Unless its 
multivalued fields and its using the UninvertedField - not sure how the 
filterCache plays into that.

- Mark

Grant Ingersoll wrote:
>
> On Jun 17, 2009, at 10:32 PM, Mark Miller wrote:
>
>> Right, so if you are on 1.3 or early 1.4 dev, with so many uniques, 
>> you should be using the FieldCache method of faceting. The RAM 
>> depends on the number of documents and number of uniques terms mostly.
>>
>> With 1.4 you may be using an UninvertedField though (are your facet 
>> fields multivalued or tokenized?), and I know much less about that.
>>
>> I'd try a cache size of 10,000 and see how it goes.
>
> I'm not so sure about that, my guess is your going to get hammered on 
> garbage collection when you do commits with something that big.
>
> Let's take a step back.  These are LRU cache's, the fact that you have 
> a zero hit ratio does not mean caching isn't working or that you 
> necessarily need a bigger cache.  It suggests to me that your 
> application is not the type that can benefit from caching of 
> filters.   My understanding is that in certain cases with the new 1.4 
> faceting, it ends up using the filterCache as well.  I believe the 
> admin will give stats on the number of big terms, etc.
>
> Perhaps you can give a bit more detail about your application and why 
> you think that cache ratio is causing slower faceting.
>
> Have you actually done some profiling/timings on the faceting?
>
>
>>
>> - Mark
>>
>> Manepalli, Kalyan wrote:
>>> Got that, if its number of cache entries, definitely its very low. I 
>>> have around 10,000 unique items to facet on. Does the RAM size 
>>> depend on Document size.
>>>
>>> Thanks,
>>> Kalyan Manepalli
>>> -----Original Message-----
>>> From: Mark Miller [mailto:markrmiller@gmail.com]
>>> Sent: Wednesday, June 17, 2009 7:13 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: FilterCache issue
>>>
>>> Its been a while since I've thought about this sort of thing, but it
>>> looks like your cache is way too small and things get evicted before
>>> being used. How many uniques are you faceting on? 512 is the number of
>>> cache entries, not the size in kb/mb.
>>>
>>> Try raising it - perhaps a lot ;) But consider that you have to have 
>>> the
>>> RAM to accommodate as well ...
>>>
>>> What version of Solr are you using?
>>>
>>> -- 
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>> Manepalli, Kalyan wrote:
>>>
>>>> Hi,
>>>>            I am seeing an issue with the filtercache setting on my 
>>>> solr app which is causing slower faceting.
>>>>
>>>> Here is the configuration.
>>>> <filterCache class="solr.LRUCache" size="512" initialSize="512" 
>>>> autowarmCount="256"/>
>>>>
>>>> Statistics:
>>>> description:  LRU Cache(maxSize=512, initialSize=512, 
>>>> autowarmCount=256, 
>>>> regenerator=org.apache.solr.search.SolrIndexSearcher$1@8d41f2)
>>>> stats: lookups : 979692
>>>> hits : 6904
>>>> hitratio : 0.00
>>>> inserts : 973531
>>>> evictions : 972978
>>>> size : 512
>>>> warmupTime : 1479
>>>> cumulative_lookups : 61660491
>>>> cumulative_hits : 516057
>>>> cumulative_hitratio : 0.00
>>>> cumulative_inserts : 61170111
>>>> cumulative_evictions : 61153787
>>>>
>>>> As we can see the cache hit ratio is almost zero. How do I improve 
>>>> the filter cache.
>>>> Also wanted to know what does the size mean. Is it number of 
>>>> documents or the memory size (kb/mb)
>>>>
>>>> Any suggestions in this regard will be very helpful.
>>>>
>>>> Thanks,
>>>> Kalyan Manepalli
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> -- 
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) 
> using Solr/Lucene:
> http://www.lucidimagination.com/search
>


-- 
- Mark

http://www.lucidimagination.com




Mime
View raw message