incubator-lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From goran kent <gorank...@gmail.com>
Subject Re: [lucy-user] Managing fresh/stale indexes
Date Wed, 02 Nov 2011 07:29:28 GMT
Thanks to all for responding.

OK, to summarise (and in the interests of keeping things simple):

merging/updating indexers:
-------------------------------------

  touch $idx_lockfile
  rm -rf $index
  mv $fresh $index
  unlink $idx_lockfile
  touch $idx_has_changed
  # ie, don't bother with symlinks and whatnot since active searchers
have cached everything.

searchers:
--------------

loop
  $lockfound = 0;
  while (-f $idx_lockfile) {
      # pause for 100ms
      select(undef, undef, undef, 0.1); }
      $lockfound = 1;
  }

  if ($lockfound) {
      $lockfound = 0;
      # restart_searcher
      goto restart;
  }
end

As I'm typing this though, I'm thinking, what about
Lucy::Search::IndexSearcher->serve?
It's in a loop hiding it's internals - how do I get *it* to honour my
$idx_lockfile?

Hack LucyX/Remote/SearchServer.pm's sub serve()?  Perhaps between

read( $client_sock, $buf, $len );
+ sleep while $idx_lockfile is extant
my $response   = $dispatch{$method}->( $self, thaw($buf) );

My final question:  Peter says "For full index swap-outs, I just let
existing Searchers finish and re-open
themselves when they realize the underlying index is different" --
peter are you saying LucyX::Remote::SearchServer does this
automatically on it's own without intervention needed, or should I
have a wrapper with LucyX::Remote::SearchServer ensconced which
monitors the indexes and kills/restarts LucyX::Remote::SearchServer
when a changed index is detected (-f $idx_has_changed)?

urgh, this is getting dirty ugly.

/throws toys out cot, sobs

On 11/2/11, Peter Karman <peter@peknet.com> wrote:
> Marvin Humphrey wrote on 11/1/11 12:36 PM:
>> On Tue, Nov 01, 2011 at 11:37:30AM +0200, goran kent wrote:
>>> I'm pondering about the best approach to handle fresh/stale indexes
>>> while searches are happening (ie, we don't want to disrupt an active
>>> search).
>>
>> Once an IndexSearcher has opened successfully, it no longer needs access
>> to
>> the index dir[1] -- you can actually wipe the index dir and the
>> IndexSearcher
>> will keep functioning indefinitely.  This is achieved by caching open
>> filehandles to all files the IndexSearcher could ever need access to, so
>> that
>> it will still have access even if the files are deleted[2].
>>
>
> fwiw, this is exactly what I do.
>
> For incremental updates, my Searcher wrapper code[1] detects when the index
> has
> changed and re-opens itself.
>
> For full index swap-outs, I just let existing Searchers finish and re-open
> themselves when they realize the underlying index is different. When the
> open
> filehandles eventually close, then the filesystem actually removes the old
> index.
>
> [1] https://metacpan.org/module/SWISH::Prog::Lucy::Searcher
>
> --
> Peter Karman  .  http://peknet.com/  .  peter@peknet.com
>

Mime
View raw message