lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Erick Erickson (Commented) (JIRA)" <>
Subject [jira] [Commented] (SOLR-3221) Make Shard handler threadpool configurable
Date Fri, 09 Mar 2012 19:18:58 GMT


Erick Erickson commented on SOLR-3221:

The 3x patch runs through all the tests, but the trunk fails.

The first one at least (and I'd guess the others as well) apparently don't depend on this
seed, they fail in IntelliJ too. They all seem to fail on SearchHandler[298]. I think once
it was a result of something not being implemented in one of the Mocks, but can't swear to

Errors in the debugger like: java.lang.IllegalArgumentException: Invalid uri 'http://://[::1]:33332/solr/select':
escaped absolute path not valid. What????

I don't have time to look into it now...

ant test -Dtestcase=TestDistributedGrouping -Dtestmethod=testDistribSearch -Dtests.seed=-7e47ec572525db2b:747a1a2f9e04dca3:611098fa6c7bdbe4
ant test -Dtestcase=FullSolrCloudDistribCmdsTest -Dtestmethod=testDistribSearch -Dtests.seed=666cf0bc1fb2bbe2:-307010e33b27cd24:611098fa6c7bdbe4
ant test -Dtestcase=DistributedSpellCheckComponentTest -Dtestmethod=testDistribSearch -Dtests.seed=-19364590f7798def:3cfbf5694891d7b:611098fa6c7bdbe4
ant test -Dtestcase=TestDistributedSearch -Dtestmethod=testDistribSearch -Dtests.seed=6cdbfdcabaf8471c:af7365c8c8b063:-35863e1b26976412
ant test -Dtestcase=OverseerTest -Dtestmethod=testShardLeaderChange -Dtests.seed=47a8a04f4df29e39:7cf1cab9cde30f9f:-35863e1b26976412
ant test -Dtestcase=BasicDistributedZkTest -Dtestmethod=testDistribSearch -Dtests.seed=-37854d406161a603:-3d7c63121a0b99fe:-a77be13ddf887d7
ant test -Dtestcase=DistributedTermsComponentTest -Dtestmethod=testDistribSearch -Dtests.seed=6fe8b3c1860d0a59:-20d9908f662d6a90:-35863e1b26976412

> Make Shard handler threadpool configurable
> ------------------------------------------
>                 Key: SOLR-3221
>                 URL:
>             Project: Solr
>          Issue Type: Improvement
>    Affects Versions: 3.6, 4.0
>            Reporter: Greg Bowyer
>            Assignee: Erick Erickson
>              Labels: distributed, http, shard
>         Attachments: SOLR-3221-3x_branch.patch, SOLR-3221-3x_branch.patch, SOLR-3221-3x_branch.patch,
SOLR-3221-3x_branch.patch, SOLR-3221-trunk.patch, SOLR-3221-trunk.patch, SOLR-3221-trunk.patch,
> From profiling of monitor contention, as well as observations of the
> 95th and 99th response times for nodes that perform distributed search
> (or ‟aggregator‟ nodes) it would appear that the HttpShardHandler code
> currently does a suboptimal job of managing outgoing shard level
> requests.
> Presently the code contained within lucene 3.5's SearchHandler and
> Lucene trunk / 3x's ShardHandlerFactory create arbitrary threads in
> order to service distributed search requests. This is done presently to
> limit the size of the threadpool such that it does not consume resources
> in deployment configurations that do not use distributed search.
> This unfortunately has two impacts on the response time if the node
> coordinating the distribution is under high load.
> The usage of the MaxConnectionsPerHost configuration option results in
> aggressive activity on semaphores within HttpCommons, it has been
> observed that the aggregator can have a response time far greater than
> that of the searchers. The above monitor contention would appear to
> suggest that in some cases its possible for liveness issues to occur and
> for simple queries to be starved of resources simply due to a lack of
> attention from the viewpoint of context switching.
> With, as mentioned above the http commons connection being hotly
> contended
> The fair, queue based configuration eliminates this, at the cost of
> throughput.
> This patch aims to make the threadpool largely configurable allowing for
> those using solr to choose the throughput vs latency balance they
> desire.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message