lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Wellnhofer <wellnho...@aevum.de>
Subject Re: [lucy-user] Perfomance issue
Date Wed, 05 Aug 2015 12:55:57 GMT
On 05/08/2015 06:36, richter@ecos.de wrote:
> I have two questions:
>
> 1.) As far as I see I have to commit and recreate the indexer every time when I have
made changes, otherwise the changes will not be seen by the other processes (or even the process
itself). On the other side, I have to destroy and recreate the SearchIndexer to see the new
documents in the index.

It's enough to call "commit" on an Indexer to make changes visible. You 
shouldn't recreate an Indexer immediately after comitting because every new 
Indexer holds a write lock on the index until it's committed. Just create a 
Indexer before adding documents, then call commit and destroy it.

Since you mention "destroy" explicitly, are you using the C bindings or the 
Perl bindings?

> While searching itself takes only 10-30 ms. The process of destroy/commit and recreate
takes up to 400ms. This makes things slow.

How many documents do you add in an indexing run? Adding only a few small 
documents should typically be faster than 400ms, but sometimes, it can take 
longer if some larger segments have to be merged. See the FastUpdates guide in 
the Lucy cookbook for how to make updates consistently fast:

     https://metacpan.org/pod/Lucy::Docs::Cookbook::FastUpdates

> 2.) From time to time I have to restart the process that heavily uses the SearchIndexer.
Searching gets very slow (up 10-60 seconds, instead of milliseconds). Simply restarting the
process fixes this, so it's not an issue on how the index is organized on disk. Any idea how
to track down this?

First, I'd try to find out which call into Lucy takes so long, whether the 
process is consuming CPU the whole time, and how the overall memory behavior 
of the process looks like. If the process is hanging for multiple seconds, you 
could also try to attach a debugger to the running process and see where it hangs.

Does this process only use IndexSearcher or does it also use Indexer? If 
there's an uncommitted Indexer, it might be a locking issue. But you'd 
probably get a lock timeout error in this case.

Nick


Mime
View raw message