lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Jelsma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-4509) Disable Stale Check - Distributed Search (Performance)
Date Thu, 28 Feb 2013 14:29:13 GMT

    [ https://issues.apache.org/jira/browse/SOLR-4509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589557#comment-13589557
] 

Markus Jelsma commented on SOLR-4509:
-------------------------------------

Hi Ryan,

Until now i've not seen real significant latency improvements when performing concurrent stress
tests. The response time still hovers between 70ms and 100ms with and without the patch using
today's trunk. I did see some disturbing exceptions, it looks like sometimes connections are
reused that are already dead, returning a connection refused exception. Sometimes shards are
also reported dead yielding the dreaded no servers hosting shard error.

Here are some exceptions:

{code}
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available
to handle this request:[...hosts here...]
        at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:333)
        at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:171)
        at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:135)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        ... 3 more
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking
to server at: http://host/solr/shard_d
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:416)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
        at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:264)
        ... 10 more
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
        at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
        at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:353)
        ... 12 more
{code}

{code}
2013-02-28 13:00:15,666 WARN [solr.cloud.RecoveryStrategy] - [main-EventThread] - : Stopping
recovery for zkNodeName=178.21.118.192:8080_solr_openindex_dcore=openindex_d
2013-02-28 13:00:17,199 WARN [solr.update.PeerSync] - [main-EventThread] - : PeerSync: core=shard_d
url=http://host/solr  couldn't connect to http://host/shard/, counting as success
2013-02-28 13:00:17,201 ERROR [solr.cloud.SyncStrategy] - [main-EventThread] - : Sync request
error: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://host/shard
2013-02-28 13:00:17,205 ERROR [solr.cloud.SyncStrategy] - [recoveryCmdExecutor-17-thread-1]
- : http://host/shard/: Could not tell a replica to recover:org.apache.solr.
client.solrj.SolrServerException: Server refused connection at: http://host
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:409)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
        at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:298)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://host refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:353)
        ... 5 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 12 more
{code}

I'm not sure yet why these occur, maybe it's my Tomcat settings that do not match Jetty's
defaults.
                
> Disable Stale Check - Distributed Search (Performance)
> ------------------------------------------------------
>
>                 Key: SOLR-4509
>                 URL: https://issues.apache.org/jira/browse/SOLR-4509
>             Project: Solr
>          Issue Type: Improvement
>          Components: search
>         Environment: 5 node SmartOS cluster (all nodes living in same global zone - i.e.
same physical machine)
>            Reporter: Ryan Zezeski
>            Priority: Minor
>         Attachments: SOLR-4509.patch
>
>
> By disabling the Apache HTTP Client stale check I've witnessed a 2-4x increase in throughput
and reduction of over 100ms.  This patch was made in the context of a project I'm leading,
called Yokozuna, which relies on distributed search.
> Here's the patch on Yokozuna: https://github.com/rzezeski/yokozuna/pull/26
> Here's a write-up I did on my findings: http://www.zinascii.com/2013/solr-distributed-search-and-the-stale-check.html
> I'm happy to answer any questions or make changes to the patch to make it acceptable.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message