incubator-lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Kurz <n...@verse.com>
Subject Re: [lucy-user] Concurrent searching
Date Mon, 14 Nov 2011 22:58:49 GMT
On Mon, Nov 14, 2011 at 5:15 AM, Marvin Humphrey <marvin@rectangular.com> wrote:
> On Mon, Nov 14, 2011 at 12:59:30PM +0200, goran kent wrote:
>> If these two could be addressed:
>>
>> -  LucyX::Remote::SearchClient to perform remote searches in parallel
>> as opposed to serially, and
>
> As of this moment (r1201554), ClusterSearcher's interface, documentation, and
> result aggregation logic are done.

Wow Marvin, that's fabulous!

>> -  LucyX::Remote::SearchServer to fork on each new client or otherwise
>> allow multiple search clients at once (ie, typical TCP/IP
>> client/server behaviour)
>
> Hacking fork() into the SearchServer#serve loop is probably the logical choice
> to solve your immediate problem -- we aren't giving you much to work with,
> after all!  But hiding a fork() call in a library function can't be our final
> solution for Lucy -- that takes SearchServer from dubiously architected to
> laughable. :)

Laughable, and as you point out not right for Core, but it might
actually work just fine for 10-or-so of requests per second.  Fork is
fast on Linux, and we don't actually want too many concurrent requests
happening at the same time.  We're presuming that these searches are
processor bound, so it would hurt us to have too many more than we
have cores.

But rather than forking in the handler, I think it might be easier to
write a wrapper that does the fork.  You might even be able to just
configure "inetd" to launch a new server every time you get a search
request.  If that doesn't handle it, POE (as Goran mentioned earlier)
wouldn't be a bad  choice. Or perhaps
http://search.cpan.org/~rhandom/Net-Server-0.94/lib/Net/Server.pm.

If you use one of those two,  you might as well prefork.  The idea is
that you fire off a bunch of servers, all listening on the same port,
and they fight to see who accepts first.  You need a manager process
that makes sure you have the right number of children, and to restart
them in case they die.  It's very smooth, and should work out of the
box with Marvin's code.

Good luck (he calls from the sidelines while Marvin does the real work),

--nate

Mime
View raw message