lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkadi Colson <ark...@smartbit.be>
Subject Re: Replication Index fetch failed
Date Mon, 10 Oct 2016 13:15:43 GMT
Hi

I could not find "Could not download file" in the logs. Should I 
increase the log level somewhere? Just let me know... so I can provide 
you more detailed logs...

Thx!
Arkadi


On 02-09-16 11:21, Arkadi Colson wrote:
> Hi
>
> I cannot find a string in the logs matching "Could not download file...".
>
> This info is logged on the slave:
>
> WARN  - 2016-09-02 09:28:36.923; [c:intradesk s:shard10 r:core_node23 
> x:intradesk_shard10_replica1] 
> org.apache.solr.handler.IndexFetcher$FileFetcher; Error in fetching 
> file: _5qd6_ya.liv (downloaded 0 of 13692 bytes)
> java.io.EOFException
>         at 
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168)
>         at 
> org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160)
>         at 
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1460)
>         at 
> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1426)
>         at 
> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:852)
>         at 
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:428)
>         at 
> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
>         at 
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:388)
>         at 
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:156)
>         at 
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408)
>         at 
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
>         at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
>
> And this on the master:
>
> WARN  - 2016-09-02 09:28:36.936; [c:intradesk s:shard10 r:core_node13 
> x:intradesk_shard10_replica2] 
> org.apache.solr.handler.ReplicationHandler$DirectoryFileStream; 
> Exception while writing response for params: 
> generation=124148&qt=/replication&file=_5qd6_ya.liv&checksum=true&wt=filestream&command=filecontent&maxWriteM
> BPerSec=18.75
> java.nio.file.NoSuchFileException: 
> /var/solr/data/intradesk_shard10_replica2/data/index.20160816102332501/_5qd6_ya.liv
>         at 
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>         at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>         at 
> sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>         at 
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
>         at java.nio.channels.FileChannel.open(FileChannel.java:287)
>         at java.nio.channels.FileChannel.open(FileChannel.java:335)
>         at 
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
>         at 
> org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:192)
>         at 
> org.apache.solr.handler.ReplicationHandler$DirectoryFileStream.write(ReplicationHandler.java:1435)
>         at org.apache.solr.core.SolrCore$3.write(SolrCore.java:2154)
>         at 
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:49)
>         at 
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:731)
>         at 
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:473)
>         at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
>         at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
>         at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at 
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at 
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Thread.java:745)
>
> Or should I enable more logging somewhere? FYI, the file 
> /var/solr/data/intradesk_shard10_replica2/data/index.20160816102332501/_5qd6_ya.liv 
> does NOT exist on the master.
>
> root@solr29-gs:~# ls 
> /var/solr/data/intradesk_shard10_replica2/data/index.20160816102332501/ 
> | grep 5qd6
> _5qd6.fdt
> _5qd6.fdx
> _5qd6.fnm
> _5qd6_Lucene50_0.doc
> _5qd6_Lucene50_0.pos
> _5qd6_Lucene50_0.tim
> _5qd6_Lucene50_0.tip
> _5qd6.si
> _5qd6_yi.liv
>
> Thanks!
>
> Arkadi
>
>
> On 02-09-16 06:43, Shalin Shekhar Mangar wrote:
>>
>> On Thu, Sep 1, 2016 at 6:05 PM, Arkadi Colson <arkadi@smartbit.be 
>> <mailto:arkadi@smartbit.be>> wrote:
>>
>>     ERROR - 2016-09-01 14:30:43.653; [c:intradesk s:shard1
>>     r:core_node5 x:intradesk_shard1_replica1]
>>     org.apache.solr.common.SolrException; Index fetch failed
>>     :org.apache.solr.common.SolrException: Unable to download
>>     _6f46_cj.liv completely. Downloaded 0!=5596
>>         at
>> org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1554)
>>         at
>> org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1437)
>>         at
>> org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:852)
>>         at
>> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:428)
>>         at
>> org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
>>
>>
>>
>> There should be another exception in the logs that looks like the 
>> following:
>> "Could not download file"...
>>
>> That one will have a more useful stack trace. Can you please find it 
>> and paste it on email?
>>
>> -- 
>> Regards,
>> Shalin Shekhar Mangar.
>
>


Mime
View raw message