lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <gsi...@gmail.com>
Subject Re: payload queries running slow
Date Mon, 21 Dec 2009 14:37:11 GMT

On Dec 20, 2009, at 3:41 AM, Raghuveer Kancherla wrote:

> Hi Grant,
> My queries are about 5 times slower when using payloads as compared to
> queries that dont use payloads on the same index. I have not done any
> profiling yet, I am trying out lucid gaze now.

How do they compare to just doing SpanQueries?  Would be interesting to see the three:
1. "Normal" queries
2. Span Queries
3. Payloads


> I do all the load testing after warming up.
> Since my index is small ~1 GB, was wondering if a ramDirectory will help
> instead of the default Directory implementation for the indexReader?
> 

I suppose, but probably not that big of a difference on a properly warmed index.


> Thanks,
> Raghu
> 
> 
> 
> On Thu, Dec 17, 2009 at 6:58 PM, Grant Ingersoll <gsingers@apache.org>wrote:
> 
>> 
>> On Dec 17, 2009, at 4:52 AM, Raghuveer Kancherla wrote:
>> 
>>> Hi,
>>> With help from the group here, I have been able to set up a search
>>> application with payloads enabled. However, there is a noticeable
>> increase
>>> in query response times with payloads as compared to the same queries
>>> without payloads. I am also seeing a lot more disk IO (I have a 7200 rpm
>>> disk) and comparatively lesser cpu usage.
>>> 
>>> I am guessing this is because of the use of payloadTermQuery and
>>> payloadNearQuery  both of which extend SpanQuery formats. SpanQueries
>> read
>>> the positions index which will be much larger than the index accessed by
>> a
>>> simple TermQuery.
>>> 
>>> Is there any way of making this system faster without having to
>> distribute
>>> the index. My index size is hardly 1GB (~200k documents and only one
>> field
>>> to search in). I am experiencing query times as high as 2 seconds
>> (average).
>>> 
>>> Any indications on the direction in which I can experiment will also be
>> very
>>> helpful.
>>> 
>> 
>> Yeah, payloads are going to be slower, but how much slower are they for
>> you? Are you warming up those queries?
>> 
>> Also, have you done any profiling?
>> 
>> 
>>> I looked at HathiTrust digital library articles. The methods indicated
>> there
>>> talk about avoiding reading the positions index (converting PhraseQueries
>> to
>>> TermQueries). That will not work in my case because, I still have to read
>>> the positions index to get the payload information during scoring. Let me
>>> know if my understanding is incorrect.
>>> 
>>> 
>>> Thanks,
>>> -Raghu
>> 
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>> 
>> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using
>> Solr/Lucene:
>> http://www.lucidimagination.com/search
>> 
>> 


Mime
View raw message