lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Britske <gbr...@gmail.com>
Subject Re: big perf-difference between solr-server vs. SOlrJ req.process(solrserver)
Date Thu, 27 Dec 2007 16:01:34 GMT

after inspecting solrconfig.xml I see that I already have enabled lazy field
loading by: 
<enableLazyFieldLoading>true</enableLazyFieldLoading> (I guess it was
enabled by default) 

Since any query returns about 10 fields (which differ from query to query) ,
would this mean that only these 10 of about 2000-4000 fields are retrieved /
loaded? 

Thanks,
Geert-Jan 



Erick Erickson wrote:
> 
> From a Lucene perspective, it's certainly possible to do lazy field
> loading. That is, when loading a document you can determine at
> run time what fields to load, even on a per-document basis. I'm
> not entirely sure how to accomplish this in Solr, but I'd give
> long odds that there's a way.....
> 
> I did a writeup on this on the Wiki, see:
> 
> http://wiki.apache.org/lucene-java/FieldSelectorPerformance?highlight=%28fieldselectorperformance%29
> 
> 
> The title is FieldSelectorPerformance if you need to search the Wiki...
> 
> Best
> Erick
> 
> On Dec 27, 2007 10:28 AM, Britske <gbrits@gmail.com> wrote:
> 
>>
>>
>>
>> Yonik Seeley wrote:
>> >
>> > On Dec 27, 2007 9:45 AM, Britske <gbrits@gmail.com> wrote:
>> >> I am using SolrJ to communicate with SOLR. My Solr-queries perform
>> within
>> >> range (between 50 ms and 300 ms) by looking at the solr log as
>> ouputted
>> >> on
>> >> my (windows) commandline.
>> >>
>> >> However I discovered that the following command at all times takes
>> >> significantly longer than the number outputted in the solr-log,
>> >> (sometimes
>> >> about 400% longer):
>> >
>> > It's probably due to stored field retrieval.
>> > The time in the response includes everything except the time to write
>> > the response (since it appears at the beginning).  Writing the
>> > response involves reading the stored fields of documents (this was
>> > done to allow one to stream a large number of documents w/o having
>> > them all in memory).
>> >
>> > SolrJ's parsing of the response should be a relatively small constant
>> > cost.
>> >
>> > -Yonik
>> >
>> >
>>
>> Is it normal to see this much time taken in stored field retrieval? And
>> where would I start to make sure that it is indeed caused by stored field
>> retrieval?
>>
>> It seems quite much to me, although I have kind if an out of the ordinary
>> setup with between 2000-4000 stored fields per document. By far the
>> largest
>> part is taken by various 'product-variants' and their respective prices
>> (indexed field) and other characteristics (stored only).
>> However only about 10 stored fields per document are returned for any
>> possible query.
>>
>> Would the time taken still include iterating the non-returned fields (of
>> which there are many in my case), or are only the returned fields
>> retrieved
>> in a map-like implementation?
>>
>> Thanks,
>> Geert-Jan
>> --
>> View this message in context:
>> http://www.nabble.com/big-perf-difference-between-solr-server-vs.--SOlrJ-req.process%28solrserver%29-tp14513964p14514441.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 

-- 
View this message in context: http://www.nabble.com/big-perf-difference-between-solr-server-vs.--SOlrJ-req.process%28solrserver%29-tp14513964p14514852.html
Sent from the Solr - User mailing list archive at Nabble.com.


Mime
View raw message