lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dag Lem <...@nimrod.no>
Subject Re: [lucy-user] SearchServer / ClusterSearcher - massive performance hit
Date Thu, 25 Oct 2012 19:56:18 GMT
Nathan Kurz <nate@verse.com> writes:

> On Wed, Oct 24, 2012 at 5:08 AM, Dag Lem <dag@nimrod.no> wrote:
> > 1. Get rid of as many network roundtrips as possible.
> > 2. Design a (simple) custom application protocol, to get rid of the
> >    overhead of Storable.
> 
> Hi Dag --
> 
> The profiling is great, and your conclusions are well founded.
> 
> There's a strong desire to keep the core simple and free of outside
> dependencies, but done as a LucyX extension you could leverage some
> existing libraries.  There has been some previous discussion of this
> on the dev list:
> http://lucene.472066.n3.nabble.com/lucy-dev-ClusterSearcher-td3483025.html
> 
> You'll find a lot of support on this list if you want to take this on.
> 
> --nate
> 

Hi Nathan,

It's good to hear that you appreciate the feedback!

I have now browsed through the previous discussion. Some questions /
comments:

Is the data which needs to be transferred over the network complex
enough to warrant a general serializer/deserializer mechanism? My
first instinct would be to come up with a custom application protocol
(exchanging only relevant data, not full objects), but that may be
highly impractical for all I know. With a custom protocol, you should
also be reasonably happy with the high level design first, to avoid
costly rewrites. No dependencies, though :-)

I don't have any experience with the many solutions for network
communication. However if, for the sake of argument, someone
implemented a solution using libev (using the Perl EV module?), why
would it be problematic to include it as a dependency for Lucy, as
Marvin stated?

In any case, I believe it's a good idea to first sort out any high(er)
level issues which can impede performance on a more fundamental level
(like network roundtrips).

I don't have any Lucy-fu (nor much time, unfortunately) to take on any
of the fundamental issues.

Perhaps I could conceivably look into something like EV at some
unspecified later time, since replacing the networking layer shouldn't
impact on anything else. I'm not so sure I'd be the best man for that
job, though...

-- 
Best regards,

Dag Lem

Mime
View raw message