lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Jelsma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-4117) IO error while trying to get the size of the Directory
Date Wed, 28 Nov 2012 14:52:59 GMT

    [ https://issues.apache.org/jira/browse/SOLR-4117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505507#comment-13505507
] 

Markus Jelsma commented on SOLR-4117:
-------------------------------------

I have another node now logging the same exception for a core that has 0 docs which is not
the leader but clusterstate says the node is active and does not attempt recovery. To my surprise
it has two index.NUMBER directories of different sizes and index.properties points to the
largest directory.

The node won't come back up properly. Search and indexing works but accessing the GUI is impossible:
{code}
2012-11-28 14:50:00,026 ERROR [solr.servlet.SolrDispatchFilter] - [http-8080-exec-6] - : null:org.apache.solr.common.SolrException:
Error handling 'status' action 
        at org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:724)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:157)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
        at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:372)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
        at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
        at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.common.SolrException: java.util.concurrent.RejectedExecutionException
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1674)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1330)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1265)
        at org.apache.solr.handler.admin.CoreAdminHandler.getCoreStatus(CoreAdminHandler.java:996)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleStatusAction(CoreAdminHandler.java:710)
        ... 18 more
Caused by: java.util.concurrent.RejectedExecutionException
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:92)
        at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:603)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1605)
        ... 22 more
{code}


                
> IO error while trying to get the size of the Directory
> ------------------------------------------------------
>
>                 Key: SOLR-4117
>                 URL: https://issues.apache.org/jira/browse/SOLR-4117
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud
>    Affects Versions: 5.0
>         Environment: 5.0.0.2012.11.28.10.42.06
> Debian Squeeze, Tomcat 6, Sun Java 6, 10 nodes, 10 shards, rep. factor 2.
>            Reporter: Markus Jelsma
>            Assignee: Mark Miller
>            Priority: Minor
>             Fix For: 5.0
>
>
> With SOLR-4032 fixed we see other issues when randomly taking down nodes (nicely via
tomcat restart) while indexing a few million web pages from Hadoop. We do make sure that at
least one node is up for a shard but due to recovery issues it may not be live.
> One node seems to work but generates IO errors in the log and ZookeeperExeption in the
GUI. In the GUI we only see:
> {code}
> SolrCore Initialization Failures
>     openindex_f: org.apache.solr.common.cloud.ZooKeeperException:org.apache.solr.common.cloud.ZooKeeperException:

> Please check your logs for more information
> {code}
> and in the log we only see the following exception:
> {code}
> 2012-11-28 11:47:26,652 ERROR [solr.handler.ReplicationHandler] - [http-8080-exec-28]
- : IO error while trying to get the size of the Directory:org.apache.lucene.store.NoSuchDirectoryException:
directory '/opt/solr/cores/shard_f/data/index' does not exist
>         at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:217)
>         at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:240)
>         at org.apache.lucene.store.NRTCachingDirectory.listAll(NRTCachingDirectory.java:132)
>         at org.apache.solr.core.DirectoryFactory.sizeOfDirectory(DirectoryFactory.java:146)
>         at org.apache.solr.handler.ReplicationHandler.getIndexSize(ReplicationHandler.java:472)
>         at org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:568)
>         at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:213)
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
>         at org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:240)
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1830)
>         at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:476)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
>         at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
>         at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:744)
>         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2274)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message