lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gupta, Rajiv" <Rajiv.Gu...@netapp.com>
Subject RE: [lucy-user] Speed up Search with Lucy::Search::IndexSearcher and Lucy::Search::PolySearcher from multiple index folders
Date Thu, 22 Sep 2016 07:02:30 GMT
Hi Nick, 

Thanks for your reply! The another constraint with the single index approach is our index
locations are dynamic and search happened through APIs which construct the index location
based on certain input parameters. 

However, I did modification in code to do fork process to complete the actions. In that process
Searcher object gets created and destroyed and I get all search hits in an array that I process
later. I did not use polysercher. This increases the speed. 

Thanks,
Rajiv Gupta

-----Original Message-----
From: Nick Wellnhofer [mailto:wellnhofer@aevum.de] 
Sent: Friday, September 16, 2016 3:51 PM
To: user@lucy.apache.org
Subject: Re: [lucy-user] Speed up Search with Lucy::Search::IndexSearcher and Lucy::Search::PolySearcher
from multiple index folders

On 14/09/2016 09:05, Gupta, Rajiv wrote:
> I'm creating indexes on multiple subfolders under one parent folder.
>
> Indexes are created on multiple folders since files are getting created in parallel and
I want to avoid segment locking between multiple indexers.

> I did profiling using Devel::NYTProf<https://metacpan.org/pod/Devel::NYTProf> and
found two places where the maximum time was taken:
> 1.    While scanning the directory. (This I will try to solve by generating a list of
directories while the application is generating the indexes).
> 2.    When creating the searchers using Lucy::Search::IndexSearcher. This takes maximum
time when running in loop for all indexed directories.

It sounds like you're working with an excessively large number of indices. 
Maybe you should simply rethink your approach and use a single index? If you're concerned
about locking maybe a separate indexing process with some kind of notification mechanism would
help?

Nick


Mime
View raw message