lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dag Lem <>
Subject Re: [lucy-user] SearchServer / ClusterSearcher - massive performance hit
Date Tue, 23 Oct 2012 09:50:39 GMT
Marvin Humphrey <> writes:


> For instance, it is perfectly acceptable if queries which return a single hit
> are slower under ClusterSearcher than under a single local IndexSearcher.
> Single-hit queries tend to be dominated by per-query overhead rather than
> per-hit search processing, and the per-query cost of ClusterSearcher is much
> higher than that of IndexSearcher.  A tenfold degradation is not
> inconcievable.

Hmmm... I must be missing something fundamental here. Given a query
which returns a single hit I'd expect the per-query overhead to be
more or less fixed and dominated by the communication between the
client and the server(s). However I have now done further tests with
more complex queries (returning a single hit), with the expectation
that this would narrow down the percent-wise overhead of sharding, and
much to my surprise it turns out that the overhead scales with the
complexity of the query!?!

Below are approximate running times (real/user) for the test program I
attached earlier. I split the sharding test in two programs
(client+server) in order to time the parts independently.

                                                single index    single shard
"fornavn:(dag) AND etternavn:(lem)"		3.5s/3.5s	28s/(13s+14s)
"fornavn:(lem dag) AND etternavn:(lem dag)"	5.0s/5.0s	47s/(20s+24s)

For sharding, the idle times (real - user) seem to indicate that the
server waits almost as much on the client as vice versa, and thus that
the client needs almost as much resources as the server, which is
rather surprising (there is no processing of hits in the test). Also,
the more complex query causes the user time to increase in the client
and server by 7 and 10 seconds, respectively, while only 1.5s is added
when sharding is not used. How is all of this possible?

> If the search profile of an application is dominated by such small lookup
> queries -- for instance, if you are using Lucy as a key-value store -- it
> would be best to avoid ClusterSearcher until you absolutely have to use it.
> Instead, you would want to invest in either RAM or SSDs.

The queries are quite complex IMHO; the second example above would be
representative if we added a restriction on birth date into the mix.
However the queries will return a quite limited number of hits; say,
normally between 1 and 10. The IndexSearcher queries seem to be mainly
CPU bound at this time, so adding RAM and SSDs probably won't help,


> Is your search query dominated by per-query or per-hit costs -- i.e. does it
> return quickly at the level of a single IndexSearcher?

I'm not sure I fully understand this question - hopefully I will
somehow have answered it above. With IndexSearcher, I can process
approximately 1000 hits per second, a figure which drops to
approximately 100 hits per second with SearchServer / ClusterSearcher.

> If the costs are mostly per-query, then degradation in ClusterSearcher is
> to be expected and arguably less of a concern.  (Theoretically, we might look
> into things like changing how we do object serialization if we wanted to
> improve matters.)

To me, the tests above seem to indicate that there is something more
going on than object serialization overhead. Unless, of course, the
overhead is so high that simply increasing the complexity of the query
makes a really big difference, in which case the object serialization
should IMHO definitely be looked into.

> If the query is expensive to begin with, though -- because it is dominated by
> per-hit costs -- then it would be unexpected to see ClusterSearcher perform
> poorly, and we would want to find out why.

As a novice user of Lucy I have a very limited understanding of the
issues, however at least I can say that the results above are highly
unexpected to me! ;-)

Best regards,

Dag Lem

View raw message