lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanne Grinovero <sanne.grinov...@gmail.com>
Subject Re: Performance problems with lazily loaded fields
Date Mon, 21 Mar 2011 19:15:11 GMT
2011/3/21 Brian Hurt <bhurt42@gmail.com>:
> I'm having a problem with the performance of lazily-loaded fields with
> lucene.  The basic structure of the code is that I get a set of documents
> back from a query, then iterate through them, reading most fields to collect
> fragments.  This is taking an excessively long amount of time- mostly in my
> call to getField on the document.  Each such call takes milliseconds,
> meaning a query that even returns only a few dozen items takes me hundreds
> of milliseconds to process, which is causing problems elsewhere.  So the
> question I have is if there is some way I can optimize the loading of
> fields?  I am calling the doc function on the index search with a null
> FieldSelector, but this does not seem to reduce the cost of getting fields
> (indeed, it seems to slow down the whole query processing by a significant
> factor).  Is there any help anyone can give me?

Hi,
you should not use a null FieldSelector, but create one instead, for
example the org.apache.lucene.document.MapFieldSelector,
so to make sure you're only going to initialize the fields you
actually need (assuming you need a subset).
Also you might have a look into FieldCache usage, there's an excellent
chapter on it in Lucene in Action (second edition);
this approach assumes you can dedicate some memory to cache extracted
field values in memory, and you're in a read-mostly
scenario (as filling the FieldCache might be more costly than a single
extraction: it affects all elements in the index, not your Hits only).

I've also seen that the cost of Field extraction is quite cheap on
solid state disks,
compared to rotating disks at least.

Regards,
Sanne

>
> Thanks.
>
> Brian
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message