lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From goran kent <gorank...@gmail.com>
Subject Re: [lucy-user] Managing fresh/stale indexes
Date Wed, 02 Nov 2011 07:31:24 GMT
s/wrapper/monitor_script/g

On 11/2/11, goran kent <gorankent@gmail.com> wrote:
> Thanks to all for responding.
>
> OK, to summarise (and in the interests of keeping things simple):
>
> merging/updating indexers:
> -------------------------------------
>
>   touch $idx_lockfile
>   rm -rf $index
>   mv $fresh $index
>   unlink $idx_lockfile
>   touch $idx_has_changed
>   # ie, don't bother with symlinks and whatnot since active searchers
> have cached everything.
>
> searchers:
> --------------
>
> loop
>   $lockfound = 0;
>   while (-f $idx_lockfile) {
>       # pause for 100ms
>       select(undef, undef, undef, 0.1); }
>       $lockfound = 1;
>   }
>
>   if ($lockfound) {
>       $lockfound = 0;
>       # restart_searcher
>       goto restart;
>   }
> end
>
> As I'm typing this though, I'm thinking, what about
> Lucy::Search::IndexSearcher->serve?
> It's in a loop hiding it's internals - how do I get *it* to honour my
> $idx_lockfile?
>
> Hack LucyX/Remote/SearchServer.pm's sub serve()?  Perhaps between
>
> read( $client_sock, $buf, $len );
> + sleep while $idx_lockfile is extant
> my $response   = $dispatch{$method}->( $self, thaw($buf) );
>
> My final question:  Peter says "For full index swap-outs, I just let
> existing Searchers finish and re-open
> themselves when they realize the underlying index is different" --
> peter are you saying LucyX::Remote::SearchServer does this
> automatically on it's own without intervention needed, or should I
> have a wrapper with LucyX::Remote::SearchServer ensconced which
> monitors the indexes and kills/restarts LucyX::Remote::SearchServer
> when a changed index is detected (-f $idx_has_changed)?
>
> urgh, this is getting dirty ugly.
>
> /throws toys out cot, sobs
>
> On 11/2/11, Peter Karman <peter@peknet.com> wrote:
>> Marvin Humphrey wrote on 11/1/11 12:36 PM:
>>> On Tue, Nov 01, 2011 at 11:37:30AM +0200, goran kent wrote:
>>>> I'm pondering about the best approach to handle fresh/stale indexes
>>>> while searches are happening (ie, we don't want to disrupt an active
>>>> search).
>>>
>>> Once an IndexSearcher has opened successfully, it no longer needs access
>>> to
>>> the index dir[1] -- you can actually wipe the index dir and the
>>> IndexSearcher
>>> will keep functioning indefinitely.  This is achieved by caching open
>>> filehandles to all files the IndexSearcher could ever need access to, so
>>> that
>>> it will still have access even if the files are deleted[2].
>>>
>>
>> fwiw, this is exactly what I do.
>>
>> For incremental updates, my Searcher wrapper code[1] detects when the
>> index
>> has
>> changed and re-opens itself.
>>
>> For full index swap-outs, I just let existing Searchers finish and
>> re-open
>> themselves when they realize the underlying index is different. When the
>> open
>> filehandles eventually close, then the filesystem actually removes the
>> old
>> index.
>>
>> [1] https://metacpan.org/module/SWISH::Prog::Lucy::Searcher
>>
>> --
>> Peter Karman  .  http://peknet.com/  .  peter@peknet.com
>>
>

Mime
View raw message