lucy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dag Lem <...@nimrod.no>
Subject [lucy-user] SearchServer / ClusterSearcher - massive performance hit
Date Mon, 22 Oct 2012 15:12:01 GMT
I've started playing around a bit with Lucy, and I have to say it's
really, really nice!

However I've run into a problem trying to increase performance using
sharding with SearchServer / ClusterSearcher. In my tests, I get close
to a tenfold *drop* in performance using a few local shards (say, 3
shards on a 4 core server).

I have attached a small test program, which only uses one shard to
demonstrate the problem. Provided an available index and appropriate
modification to the hard coded query, the program can be run as
./test_shard.pl to test SearchServer / ClusterSearcher, and as
./test_shard.pl 0 to test IndexSearcher on the same index. BTW, my
index contains about 5 million documents.

While I would expect some overhead using SearchServer / ClusterSearcher,
the close to tenfold increase in search time I experience does seem
rather excessive. I'd need an exorbitant amount of shards just to get
the same performance as by using a single index, if I'd ever get there...

So, I'm wondering whether I'm just doing something really stupid, or
whether I've run into a problem in Lucy. Help!? :-)

If there is anything I can do to help isolate any possible problem,
please do tell me so (e.g. strace / perl profiling / ...)

In case that hase any relevance, the test system is a (really old)
RedHat 5.8 x86_64 system with two dual-core AMD Opteron 2218
processors and 4GB RAM.

-- 
Best regards,

Dag Lem


Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message