lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lansing <halim.cha...@wantedanalytics.com>
Subject Error when restarting solr servers
Date Thu, 30 Jan 2014 15:26:36 GMT
Hello,
Running solr cloud with 2 collections 5 shards and 3 replicas for each
collection, 5 zookeeper instance.
solr-4.6.0
apache-tomcat-7.0.39
zookeeper-3.4.5
jre1.7.0_21

When I try to restart a solr servers in my solr cloud I am receiving this
errors :

1861449 [localhost-startStop-1-EventThread] INFO 
org.apache.solr.cloud.ShardLeaderElectionContext  â Running the leader
process for shard shard1
1861451 [localhost-startStop-1-EventThread] INFO 
org.apache.solr.cloud.ShardLeaderElectionContext  â Checking if I should try
and be the leader.
1861451 [localhost-startStop-1-EventThread] INFO 
org.apache.solr.cloud.ShardLeaderElectionContext  â My last published State
was down, I won't be the leader.
1861451 [localhost-startStop-1-EventThread] INFO 
org.apache.solr.cloud.ShardLeaderElectionContext  â There may be a better
leader candidate than us - going back into recovery
1861452 [localhost-startStop-1-EventThread] INFO 
org.apache.solr.update.DefaultSolrCoreState  â Running recovery - first
canceling any ongoing recovery
1861452 [localhost-startStop-1-EventThread] WARN 
org.apache.solr.cloud.RecoveryStrategy  â Stopping recovery for
zkNodeName=core_node3core=Current1_shard1_replica3
1862223 [RecoveryThread] INFO  org.apache.solr.cloud.RecoveryStrategy  â
Finished recovery process. core=Current1_shard1_replica3
1862223 [RecoveryThread] INFO  org.apache.solr.cloud.RecoveryStrategy  â
Starting recovery process.  core=Current1_shard1_replica3
recoveringAfterStartup=false
1862223 [RecoveryThread] ERROR org.apache.solr.update.UpdateLog  â Exception
reading versions from log
java.nio.channels.ClosedChannelException
        at sun.nio.ch.FileChannelImpl.ensureOpen(Unknown Source)
        at sun.nio.ch.FileChannelImpl.read(Unknown Source)
        at
org.apache.solr.update.ChannelFastInputStream.readWrappedStream(TransactionLog.java:778)
        at
org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:89)
        at
org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:71)
        at
org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216)
        at
org.apache.solr.update.TransactionLog$FSReverseReader.<init>(TransactionLog.java:696)
        at
org.apache.solr.update.TransactionLog.getReverseReader(TransactionLog.java:575)
        at
org.apache.solr.update.UpdateLog$RecentUpdates.update(UpdateLog.java:942)
        at
org.apache.solr.update.UpdateLog$RecentUpdates.access$000(UpdateLog.java:885)
        at
org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1042)
        at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:280)
        at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244)
1862223 [RecoveryThread] ERROR org.apache.solr.cloud.RecoveryStrategy  â
Error while trying to recover.
core=Current1_shard1_replica3:org.apache.solr.common.SolrException: Cloud
state sti
ll says we are leader.
        at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:354)
        at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244)

1862224 [RecoveryThread] ERRwwwwOR org.apache.solr.cloud.RecoveryStrategy  â
Recovery failed - trying again... (0) core=Current1_shard1_replica3
1862224 [RecoveryThread] INFO  org.apache.solr.cloud.RecoveryStrategy  â
Wait 2.0 seconds before trying to recover again (1)
1862541 [localhost-startStop-1-SendThread(10.0.5.230:2281)] WARN 
org.apache.zookeeper.ClientCnxn  â Session 0x542fd3f2be100e6 for server
10.0.5.230/10.0.5.230:2281, unexpected error, clo
sing socket connection and attempting reconnect
java.io.IOException: Packet len11106511 is out of range!
        at
org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112)
        at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79)
        at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
1862641 [localhost-startStop-1-EventThread] INFO 
org.apache.solr.cloud.DistributedQueue  â Watcher fired on path: null state:
Disconnected type None
1862641 [localhost-startStop-1-EventThread] INFO 
org.apache.solr.cloud.DistributedQueue  â Watcher fired on path: null state:
Disconnected type None
1862641 [localhost-startStop-1-EventThread] INFO 
org.apache.solr.cloud.DistributedQueue  â Watcher fired on path: null state:
Disconnected type None
1862641 [localhost-startStop-1-EventThread] INFO 
org.apache.solr.cloud.DistributedQueue  â Watcher fired on path: null state:
Disconnected type None


..............


1270268 [http-bio-8201-exec-26] INFO 
org.apache.solr.handler.admin.CoreAdminHandler  â Going to wait for
coreNodeName: core_node10, state: recovering, checkLive: true, onlyIfLeader:
true
1270268 [http-bio-8201-exec-10] INFO 
org.apache.solr.handler.admin.CoreAdminHandler  â Going to wait for
coreNodeName: core_node11, state: recovering, checkLive: true, onlyIfLeader:
true
1270268 [http-bio-8201-exec-26] ERROR org.apache.solr.core.SolrCore  â
org.apache.solr.common.SolrException: We are not the leader
        at
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:873)
        at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:198)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
        at java.lang.Thread.run(Unknown Source)



..............



1778261 [http-bio-8201-exec-1] INFO 
org.apache.solr.handler.admin.CoreAdminHandler  â Going to wait for
coreNodeName: core_node15, state: recovering, checkLive: true, onlyIfLeader:
true
1778261 [http-bio-8201-exec-1] ERROR org.apache.solr.core.SolrCore  â
org.apache.solr.common.SolrException: We are not the leader
        at
org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:873)
        at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:198)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
        at java.lang.Thread.run(Unknown Source)

1778261 [http-bio-8201-exec-1] INFO 
org.apache.solr.servlet.SolrDispatchFilter  â [admin] webapp=null
path=/admin/cores
params={coreNodeName=core_node15&state=recovering&nodeName=10.0.5.
230:8201_solr&action=PREPRECOVERY&checkLive=true&core=Current1_shard5_replica1&wt=javabin&onlyIfLeader=true&version=2}
status=400 QTime=0
1780238 [http-bio-8201-exec-29] ERROR org.apache.solr.core.SolrCore  â
org.apache.solr.common.SolrException: no servers hosting shard:
        at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:148)
        at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
        at java.lang.Thread.run(Unknown Source)



.............


2260241 [http-bio-8201-exec-32] ERROR org.apache.solr.core.SolrCore  â
org.apache.solr.common.SolrException: Error opening new searcher
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1346)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1281)
        at
org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:96)
        at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:237)
        at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
        at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:710)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
        at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009)
        at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
        at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.lucene.store.AlreadyClosedException: Already closed
        at
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:340)
        at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:251)
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1372)
        ... 26 more




--
View this message in context: http://lucene.472066.n3.nabble.com/Error-when-restarting-solr-servers-tp4114461.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message