lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Re: [lucy-user] Concurrent searching
Date Mon, 14 Nov 2011 13:15:16 GMT
On Mon, Nov 14, 2011 at 12:59:30PM +0200, goran kent wrote:
> If these two could be addressed:
> -  LucyX::Remote::SearchClient to perform remote searches in parallel
> as opposed to serially, and

As of this moment (r1201554), ClusterSearcher's interface, documentation, and
result aggregation logic are done.  The internals are not yet complete, but it
should be an improvement over the PolySearcher/SearchClient combo.  At present
it sends requests to all remote nodes before trying to retrieve the response from
any, allowing the remotes to do their work in parallel -- that's better than
PolySearcher, which had to wait on each remote's response before sending the
next request.

In my local checkout, I've got a ClusterSearcher select() loop and
non-blocking socket i/o working which should theoretically allow the boss node
to do more while waiting for responses from the remotes, and potentially to
deal with timeouts and failovers at some point in the future.  I'm not done
tuning that yet, but you don't have to wait on it -- you can start
test-driving ClusterSearcher now if you like.
> -  LucyX::Remote::SearchServer to fork on each new client or otherwise
> allow multiple search clients at once (ie, typical TCP/IP
> client/server behaviour)

Hacking fork() into the SearchServer#serve loop is probably the logical choice
to solve your immediate problem -- we aren't giving you much to work with,
after all!  But hiding a fork() call in a library function can't be our final
solution for Lucy -- that takes SearchServer from dubiously architected to
laughable. :) 

I'm trying to figure out how to refactor SearchServer so that it continues to
encapsulate the protocol used for communicating with ClusterSearcher and
SearchClient, and continues to handle the socket communication and the
select() logic, but moves other logic to userland allowing forking servers,
preforked servers, etc.  Perhaps split up serve() into incoming() and
handle_request(), allowing userland code like this?

    my $searcher = Lucy::Search::IndexSearcher->new(index => $path);
    my $search_server = LucyX::Remote::SearchServer->new(
        port     => 7890,
        searcher => $searcher,

    while ($search_server->incoming) {
        my $pid = fork();

Marvin Humphrey

View raw message