Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of cbmcgee@ca.ibm.com designates
 32.97.182.144 as permitted sender)
In-Reply-To: <359a92830804101318l46a4a327j2552311db6126698@mail.gmail.com>
To: java-user@lucene.apache.org
MIME-Version: 1.0
Subject: Re: How to improve performance of large numbers of successive
 searches?
From: Chris McGee <cbmcgee@ca.ibm.com>
Message-ID: 
 <OFC7128742.35A91982-ON85257428.006A04E3-8525742B.004E5F10@ca.ibm.com>
Date: Mon, 14 Apr 2008 10:17:30 -0400
Content-Type: multipart/alternative;
 boundary="=_alternative 004E5E528525742B_="

--=_alternative 004E5E528525742B_=
Content-Type: text/plain; charset="US-ASCII"

Hi Erick,

Thanks for the information. I tried using a HitCollector and a 
FieldSelector. I'm getting some dramatic improvements gathering large 
result sets using the FieldSelector. As it turned out I was able to assume 
in many cases that I could break out after a specific field in each 
document.

Assuming that I need to gather all result documents each time, what are 
the advantages of using a HitCollector over Hits?

Is there some way that I can load the index portion of the lucene data 
storage into RAM without loading everything into a RAMDirectory?

Thanks,
Chris McGee


"Erick Erickson" <erickerickson@gmail.com> 
10/04/2008 04:18 PM
Please respond to
java-user@lucene.apache.org


To
java-user@lucene.apache.org
cc

Subject
Re: How to improve performance of large numbers of successive searches?


>From this <<< iterate over all of the hits>>> I infer that you're
using a Hits object. This is a no-no when getting more than 100
or so objects. In a nutshell, the query gets re-executed every 100
fetches. So your 2,000 hits are executing the query 20 times.

The Hits object is optimized for returning the top few scoring
documents rather than get the entire result set.

See HitCollector/TopDocs/TopDocCollector etc. for better ways
of doing this.

Also, if you're calling IndexReader.document(i) for each document
you'll inevitably take a lot of time as you're loading all of each 
document.
Think about lazy field loading (see FieldSelector).

Best
Erick

P.S. If this is totally off base, perhaps you could post some of the
code you think is slow....

On Thu, Apr 10, 2008 at 2:34 PM, Chris McGee <cbmcgee@ca.ibm.com> wrote:

> Hello,
>
> I am building fairly large directories (200-500 MB of disk space) using
> lucene-java. Sometimes it can take upwards of 10-15 mins to create the
> documents and write them to disk using my current configuration. I have
> upgraded to the latest 2.3.1 version and followed many of the
> recommendations offered on the wiki:
>
> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
>
> These tips have significantly improved the time to build the directory 
and
> search it. However, I have noticed that when I perform term queries 
using
> a searcher many times in rapid succession and iterate over all of the 
hits
> it can take a significant time. To perform 1000 term query searches each
> with around 2000 hits it takes well over a minute. The time seems to 
vary
> linearly based on the number of searches (ie. 10 times more searches 
take
> 10 times longer). I tried combining the searches into a BooleanQuery but
> it only shaves off a small percentage (5-10%) of the total time.
>
> I was wondering if there is a faster way to retrieve all of the results
> for my large collections of terms without using more memory and without
> taking more time to build the directory? I already looked at bypassing 
the
> searcher and using the IndexReader.termDocs() method directly to 
retrieve
> the documents but there did not seem to be much performance improvement.
> In the majority of my cases I am simplying looking for a large number of
> values to the same field. Also, I'm not interested in scoring results
> based on frequency or weights I need to retrieve all of the results
> anyway.
>
> Any help with this would be great.
>
> Thanks,
> Chris McGee


--=_alternative 004E5E528525742B_=--