Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 712A1163B4E for ; Tue, 22 Aug 2017 18:49:26 +0200 (CEST) Received: (qmail 81746 invoked by uid 500); 22 Aug 2017 16:49:24 -0000 Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: solr-user@lucene.apache.org Delivered-To: mailing list solr-user@lucene.apache.org Received: (qmail 81735 invoked by uid 99); 22 Aug 2017 16:49:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Aug 2017 16:49:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id BED5EC2FED for ; Tue, 22 Aug 2017 16:49:23 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.531 X-Spam-Level: * X-Spam-Status: No, score=1.531 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LOTSOFHASH=0.25, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, TVD_FW_GRAPHIC_NAME_MID=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id FgZ9oTsnTtWO for ; Tue, 22 Aug 2017 16:49:21 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id A583F5FBBA for ; Tue, 22 Aug 2017 16:49:20 +0000 (UTC) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id v7MGiNHS052180 for ; Tue, 22 Aug 2017 12:49:19 -0400 Received: from smtp.notes.na.collabserv.com (smtp.notes.na.collabserv.com [192.155.248.93]) by mx0b-001b2d01.pphosted.com with ESMTP id 2cgr5p918n-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 22 Aug 2017 12:49:19 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Tue, 22 Aug 2017 16:49:18 -0000 Received: from us1a3-smtp05.a3.dal06.isc4sb.com (10.146.71.159) by smtp.notes.na.collabserv.com (10.106.227.39) with smtp.notes.na.collabserv.com ESMTP; Tue, 22 Aug 2017 16:49:16 -0000 Received: from us1a3-mail70.a3.dal09.isc4sb.com ([10.142.3.73]) by us1a3-smtp05.a3.dal06.isc4sb.com with ESMTP id 2017082216491555-752130 ; Tue, 22 Aug 2017 16:49:15 +0000 MIME-Version: 1.0 In-Reply-To: Subject: Re: Few document replication not happen in solr cloud To: solr-user@lucene.apache.org From: "Sanjay Lokhande" Date: Tue, 22 Aug 2017 22:19:11 +0530 References: X-KeepSent: 613C53A1:F4E4F35F-65258184:005C4980; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1FP5 SHF190 February 24, 2016 X-LLNOutbound: False X-Disclaimed: 28387 X-TNEFEvaluated: 1 Content-type: multipart/related; Boundary="0__=EABB0B17DFCFCF108f9e8a93df938690918cEABB0B17DFCFCF10" x-cbid: 17082216-1799-0000-0000-000002ADC463 X-IBM-SpamModules-Scores: BY=0.207411; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.394815; ST=0; TS=0; UL=0; ISC=; MB=0.000000 X-IBM-SpamModules-Versions: BY=3.00007591; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000223; SDB=6.00906079; UDB=6.00454101; IPR=6.00686281; BA=6.00005550; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00016813; XFM=3.00000015; UTC=2017-08-22 16:49:16 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2017-08-22 11:55:32 - 6.00007217 x-cbparentid: 17082216-1800-0000-0000-000087948F39 Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-08-22_07:,, signatures=0 X-Proofpoint-Spam-Reason: safe --0__=EABB0B17DFCFCF108f9e8a93df938690918cEABB0B17DFCFCF10 Content-type: multipart/alternative; Boundary="1__=EABB0B17DFCFCF108f9e8a93df938690918cEABB0B17DFCFCF10" --1__=EABB0B17DFCFCF108f9e8a93df938690918cEABB0B17DFCFCF10 Content-Transfer-Encoding: quoted-printable Content-type: text/plain; charset=US-ASCII some more details on the issue. I am having 5 nodes solr cloud setup with single shard. The solr version is 5.2.1. server1 (http://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4)is the leader. A document with id '43e14a86cbdd422880cac22d9a15d3c0' was not replicated 3 nodes. Log shows that the "{add=3D[43e14a86cbdd422880cac22d9a15d3c0 (1573510697298427904)]}" request is received only by leader and server5 node. The server2, server3 & server4 node did not receive the request and hence the document is missing in these nodes. Search "43e14a86cbdd422880cac22d9a15d3c0 " C:\solrIssue\solr=5Fserver1.log INFO - 2017-07-21 05:54:59.853; [contracts shard1 core=5Fnode2 contracts=5Fshard1=5Freplica4] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica4] webapp=3D/solr path=3D/update params=3D {wt=3Djavabin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d= 75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... INFO - 2017-07-21 05:54:59.853; [contracts shard1 core=5Fnode2 contracts=5Fshard1=5Freplica4] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica4] webapp=3D/solr path=3D/update params=3D {wt=3Djavabin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d= 75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... INFO - 2017-07-21 05:59:23.845; [contracts shard1 core=5Fnode2 contracts=5Fshard1=5Freplica4] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica4] webapp=3D/solr path=3D/update params=3D {wt=3Djavabin&version=3D2} {add=3D[43e14a86cbdd422880cac22d9a15d3c0 (1573510697298427904)]} 0 26582 C:\solrIssue\solr=5Fserver2\solr.log.1 INFO - 2017-07-21 05:54:59.595; [contracts shard1 core=5Fnode4 contracts=5Fshard1=5Freplica5] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica5] webapp=3D/solr path=3D/update params=3D {update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&distrib.f= rom=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djava= bin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... INFO - 2017-07-21 05:54:59.595; [contracts shard1 core=5Fnode4 contracts=5Fshard1=5Freplica5] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica5] webapp=3D/solr path=3D/update params=3D {update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&distrib.f= rom=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djava= bin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... C:\solrIssue\solr=5Fserver3.log INFO - 2017-07-21 05:54:59.844; [contracts shard1 core=5Fnode1 contracts=5Fshard1=5Freplica3] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica3] webapp=3D/solr path=3D/update params=3D {update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&distrib.f= rom=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djava= bin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... INFO - 2017-07-21 05:54:59.844; [contracts shard1 core=5Fnode1 contracts=5Fshard1=5Freplica3] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica3] webapp=3D/solr path=3D/update params=3D {update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&distrib.f= rom=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djava= bin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... C:\solrIssue\solr=5Fserver4\solr.log.1 INFO - 2017-07-21 05:54:59.734; [contracts shard1 core=5Fnode3 contracts=5Fshard1=5Freplica1] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica1] webapp=3D/solr path=3D/update params=3D {update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&distrib.f= rom=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djava= bin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... INFO - 2017-07-21 05:54:59.734; [contracts shard1 core=5Fnode3 contracts=5Fshard1=5Freplica1] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica1] webapp=3D/solr path=3D/update params=3D {update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&distrib.f= rom=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djava= bin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... C:\solrIssue\solr=5Fserver5\solr.log.1 INFO - 2017-07-21 05:54:59.851; [contracts shard1 core=5Fnode5 contracts=5Fshard1=5Freplica2] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica2] webapp=3D/solr path=3D/update params=3D {update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&distrib.f= rom=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djava= bin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... INFO - 2017-07-21 05:54:59.851; [contracts shard1 core=5Fnode5 contracts=5Fshard1=5Freplica2] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica2] webapp=3D/solr path=3D/update params=3D {update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&distrib.f= rom=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djava= bin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad... INFO - 2017-07-21 05:59:23.843; [contracts shard1 core=5Fnode5 contracts=5Fshard1=5Freplica2] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica2] webapp=3D/solr path=3D/update params=3D {update.distrib=3DFROMLEADER&distrib.from=3Dhttp://146.XXX.com:4001/solr/co= ntracts=5Fshard1=5Freplica4/&wt=3Djavabin&version=3D2} {add=3D[43e14a86cbdd422880cac22d9a15d3c0 (1573510697298427904)]} 0 16241 There are 2 more documents which are not replicated and having similar logs entries. Can you please help me to under stand why the replication not happen on these nodes? What can be the possible reasons and how to troubleshoot the issue if reoccurs? There is no error indicating failure of replication or nodes. However there are some errors "java.util.concurrent.TimeoutException: Idle timeout expired: 50000/50000 ms". ERROR - 2017-07-21 05:59:35.181; [contracts shard1 core=5Fnode3 contracts=5Fshard1=5Freplica1] org.apache.solr.common.SolrException; null:java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 50000/50000 ms at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block (SharedBlockingCallback.java:234) at org.eclipse.jetty.server.HttpInputOverHTTP.blockForContent (HttpInputOverHTTP.java:66) at org.eclipse.jetty.server.HttpInput$1.waitForContent (HttpInput.java:476) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:121) at org.apache.solr.common.util.FastInputStream.readWrappedStream (FastInputStream.java:80) at org.apache.solr.common.util.FastInputStream.refill (FastInputStream.java:89) at org.apache.solr.common.util.FastInputStream.readByte (FastInputStream.java:207) at org.apache.solr.common.util.JavaBinCodec.readVal (JavaBinCodec.java:188) at org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument (JavaBinCodec.java:424) at org.apache.solr.common.util.JavaBinCodec.readVal (JavaBinCodec.java:245) at org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument (JavaBinCodec.java:414) at org.apache.solr.common.util.JavaBinCodec.readVal (JavaBinCodec.java:245) at org.apache.solr.common.util.JavaBinCodec.readMapEntry (JavaBinCodec.java:542) at org.apache.solr.common.util.JavaBinCodec.readVal (JavaBinCodec.java:249) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec $1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:147) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec $1.readIterator(JavaBinUpdateRequestCodec.java:135) at org.apache.solr.common.util.JavaBinCodec.readVal (JavaBinCodec.java:241) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec $1.readNamedList(JavaBinUpdateRequestCodec.java:121) at org.apache.solr.common.util.JavaBinCodec.readVal (JavaBinCodec.java:206) at org.apache.solr.common.util.JavaBinCodec.unmarshal (JavaBinCodec.java:126) at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal (JavaBinUpdateRequestCodec.java:186) at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs (JavabinLoader.java:111) at org.apache.solr.handler.loader.JavabinLoader.load (JavabinLoader.java:58) at org.apache.solr.handler.UpdateRequestHandler$1.load (UpdateRequestHandler.java:98) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody (ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest (RequestHandlerBase.java:143) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) at org.apache.solr.servlet.HttpSolrCall.execute (HttpSolrCall.java:654) at org.apache.solr.servlet.HttpSolrCall.call (HttpSolrCall.java:450) at org.apache.solr.servlet.SolrDispatchFilter.doFilter (SolrDispatchFilter.java:227) at org.apache.solr.servlet.SolrDispatchFilter.doFilter (SolrDispatchFilter.java:196) at org.eclipse.jetty.servlet.ServletHandler $CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle (ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle (SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle (SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle (ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope (ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope (SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope (ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle (ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle (ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerCollection.handle (HandlerCollection.java:110) at org.eclipse.jetty.server.handler.HandlerWrapper.handle (HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:497) at org.eclipse.jetty.server.HttpChannel.handle (HttpChannel.java:310) at org.eclipse.jetty.server.HttpConnection.onFillable (HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run (AbstractConnection.java:540) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run (QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:785) Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 50000/50000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout (IdleTimeout.java:161) at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:267) at java.util.concurrent.ScheduledThreadPoolExecutor $ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor $ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1143) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:618) ... 1 more Thanks in advance. Regards, Sanjay From: "Sanjay Lokhande" To: solr-user@lucene.apache.org Date: 08/22/2017 09:00 PM Subject: Few document replication not happen in solr cloud Hi guys, I having 5 node solr cloud setup. The logs indicates leader and 2 solr node receiving document add request. The other 2 nodes logs did not show the entry to add the document and also these nodes missing the document. - how such issue can be troubleshoot? any pointers? - any logging level which can help is this situation? Regards, Sanjay --1__=EABB0B17DFCFCF108f9e8a93df938690918cEABB0B17DFCFCF10 Content-Transfer-Encoding: quoted-printable Content-type: text/html; charset=US-ASCII Content-Disposition: inline

some more details on the issue.
=
I am having 5 nodes solr cloud setup with single sha= rd. The solr version is 5.2.1.
server1 (http://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4)is the leader.
A doc= ument with id '43e14a86cbdd422880cac22d9a15d3c0' was not replicated 3 nodes= .
Log shows that the "{add=3D[43e14a86cbd= d422880cac22d9a15d3c0 (1573510697298427904)]}" request is received onl= y by leader and server5 node.
The server2, se= rver3 & server4 node did not receive the request and hence the document= is missing in these nodes.
Search "43e14a86cbdd422880cac22d9a15d3c0 "
C:\solrIssue\solr=5Fserver1.log
INFO - 2017-07-21 05:54:59.853; [contracts shard1 core= =5Fnode2 contracts=5Fshard1=5Freplica4] org.apache.solr.update.processor.Lo= gUpdateProcessor; [contracts=5Fshard1=5Freplica4] webapp=3D/solr path=3D/up= date params=3D{wt=3Djavabin&version=3D2} {deleteByQuery=3Did:(9467353f3= 98448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89= c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7= 024ca 8ad...
INFO - 2017-07-21 05:54:59= .853; [contracts shard1 core=5Fnode2 contracts=5Fshard1=5Freplica4] org.apa= che.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica= 4] webapp=3D/solr path=3D/update params=3D{wt=3Djavabin&version=3D2} {d= eleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713= 871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0= a0af8cb24ef94d25b9691eee1f7024ca 8ad...
= INFO - 2017-07-21 05:59:23.845; [contracts shard1 core=5Fnode2 contracts= =5Fshard1=5Freplica4] org.apache.solr.update.processor.LogUpdateProcessor; = [contracts=5Fshard1=5Freplica4] webapp=3D/solr path=3D/update params=3D{wt= =3Djavabin&version=3D2} {add=3D[43e14a86cbdd422880cac22d9a15d3c0 (15735= 10697298427904)]} 0 26582
C:\solrIssue\solr=5F= server2\solr.log.1
INFO - 2017-07-21 05= :54:59.595; [contracts shard1 core=5Fnode4 contracts=5Fshard1=5Freplica5] o= rg.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Fr= eplica5] webapp=3D/solr path=3D/update params=3D{update.distrib=3DFROMLEADE= R&=5Fversion=5F=3D-1573510446380482560&distrib.from=3Dhttp://146.XX= X.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djavabin&version= =3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff414= 1a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d= 9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad...
INFO - 2017-07-21 05:54:59.595; [contracts shard1 core=5Fnode4 co= ntracts=5Fshard1=5Freplica5] org.apache.solr.update.processor.LogUpdateProc= essor; [contracts=5Fshard1=5Freplica5] webapp=3D/solr path=3D/update params= =3D{update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&am= p;distrib.from=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4= /&wt=3Djavabin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c= 261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa= 01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad= ...
C:\solrIssue\solr=5Fserver3.log
= INFO - 2017-07-21 05:54:59.844; [contracts shard1= core=5Fnode1 contracts=5Fshard1=5Freplica3] org.apache.solr.update.process= or.LogUpdateProcessor; [contracts=5Fshard1=5Freplica3] webapp=3D/solr path= =3D/update params=3D{update.distrib=3DFROMLEADER&=5Fversion=5F=3D-15735= 10446380482560&distrib.from=3Dhttp://146.XXX.com:4001/solr/contracts=5F= shard1=5Freplica4/&wt=3Djavabin&version=3D2} {deleteByQuery=3Did:(9= 467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8= 364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b96= 91eee1f7024ca 8ad...
INFO - 2017-07-21 = 05:54:59.844; [contracts shard1 core=5Fnode1 contracts=5Fshard1=5Freplica3]= org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1= =5Freplica3] webapp=3D/solr path=3D/update params=3D{update.distrib=3DFROML= EADER&=5Fversion=5F=3D-1573510446380482560&distrib.from=3Dhttp://14= 6.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djavabin&ver= sion=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7f= f4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880ca= c22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad...
C:\solrIssue\solr=5Fserver4\solr.log.1
= INFO - 2017-07-21 05:54:59.734; [contracts shard1 core=5Fnode3 con= tracts=5Fshard1=5Freplica1] org.apache.solr.update.processor.LogUpdateProce= ssor; [contracts=5Fshard1=5Freplica1] webapp=3D/solr path=3D/update params= =3D{update.distrib=3DFROMLEADER&=5Fversion=5F=3D-1573510446380482560&am= p;distrib.from=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4= /&wt=3Djavabin&version=3D2} {deleteByQuery=3Did:(9467353f398448788c= 261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa= 01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad= ...
INFO - 2017-07-21 05:54:59.734; [co= ntracts shard1 core=5Fnode3 contracts=5Fshard1=5Freplica1] org.apache.solr.= update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica1] webapp= =3D/solr path=3D/update params=3D{update.distrib=3DFROMLEADER&=5Fversio= n=5F=3D-1573510446380482560&distrib.from=3Dhttp://146.XXX.com:4001/solr= /contracts=5Fshard1=5Freplica4/&wt=3Djavabin&version=3D2} {deleteBy= Query=3Did:(9467353f398448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65= ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8c= b24ef94d25b9691eee1f7024ca 8ad...
C:\solrIssue= \solr=5Fserver5\solr.log.1
INFO - 2017-= 07-21 05:54:59.851; [contracts shard1 core=5Fnode5 contracts=5Fshard1=5Frep= lica2] org.apache.solr.update.processor.LogUpdateProcessor; [contracts=5Fsh= ard1=5Freplica2] webapp=3D/solr path=3D/update params=3D{update.distrib=3DF= ROMLEADER&=5Fversion=5F=3D-1573510446380482560&distrib.from=3Dhttp:= //146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&wt=3Djavabin&= ;version=3D2} {deleteByQuery=3Did:(9467353f398448788c261aa347d75b8b 93332ab= 7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c876aadfa01022 43e14a86cbdd4228= 80cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f7024ca 8ad...
INFO - 2017-07-21 05:54:59.851; [contracts shard1 core=5F= node5 contracts=5Fshard1=5Freplica2] org.apache.solr.update.processor.LogUp= dateProcessor; [contracts=5Fshard1=5Freplica2] webapp=3D/solr path=3D/updat= e params=3D{update.distrib=3DFROMLEADER&=5Fversion=5F=3D-15735104463804= 82560&distrib.from=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5F= replica4/&wt=3Djavabin&version=3D2} {deleteByQuery=3Did:(9467353f39= 8448788c261aa347d75b8b 93332ab7f7ff4141a371713871ab65ad 8568e0eab8364bfc89c= 876aadfa01022 43e14a86cbdd422880cac22d9a15d3c0 a0af8cb24ef94d25b9691eee1f70= 24ca 8ad...
INFO - 2017-07-21 05:59:23.= 843; [contracts shard1 core=5Fnode5 contracts=5Fshard1=5Freplica2] org.apac= he.solr.update.processor.LogUpdateProcessor; [contracts=5Fshard1=5Freplica2= ] webapp=3D/solr path=3D/update params=3D{update.distrib=3DFROMLEADER&d= istrib.from=3Dhttp://146.XXX.com:4001/solr/contracts=5Fshard1=5Freplica4/&a= mp;wt=3Djavabin&version=3D2} {add=3D[43e14a86cbdd422880cac22d9a15d3c0 (= 1573510697298427904)]} 0 16241

There are 2= more documents which are not replicated and having similar logs entries.
Can you please help me to under stand why the r= eplication not happen on these nodes? What can be the possible reasons and = how to troubleshoot the issue if reoccurs?
The= re is no error indicating failure of replication or nodes.

However there are some errors &q= uot;java.util.concurrent.TimeoutException: Idle timeout expired: 50000/5000= 0 ms".

= ERROR - 2017-07-21 05:59:35.181; [contracts shard1 core=5Fnode3 contracts= =5Fshard1=5Freplica1] org.apache.solr.common.SolrException; null:java.io.IO= Exception: java.util.concurrent.TimeoutException: Idle timeout expired: 500= 00/50000 ms
at org.eclipse.jetty= .util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)=
at org.eclipse.jetty.server.Htt= pInputOverHTTP.blockForContent(HttpInputOverHTTP.java:66)
at org.eclipse.jetty.server.HttpInput$1.waitForCo= ntent(HttpInput.java:476)
at org= .eclipse.jetty.server.HttpInput.read(HttpInput.java:121)
at org.apache.solr.common.util.FastInputStream.rea= dWrappedStream(FastInputStream.java:80)
= at org.apache.solr.common.util.FastInputStream.refill(FastInputStre= am.java:89)
at org.apache.solr.c= ommon.util.FastInputStream.readByte(FastInputStream.java:207)
at org.apache.solr.common.util.JavaBinCodec.r= eadVal(JavaBinCodec.java:188)
at= org.apache.solr.common.util.JavaBinCodec.readSolrInputDocument(JavaBinCode= c.java:424)
at org.apache.solr.c= ommon.util.JavaBinCodec.readVal(JavaBinCodec.java:245)
at org.apache.solr.common.util.JavaBinCodec.readSolr= InputDocument(JavaBinCodec.java:414)
= at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:= 245)
at org.apache.solr.common.u= til.JavaBinCodec.readMapEntry(JavaBinCodec.java:542)
at org.apache.solr.common.util.JavaBinCodec.readVal(= JavaBinCodec.java:249)
at org.ap= ache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDoc= Iterator(JavaBinUpdateRequestCodec.java:147)
= at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCo= dec$1.readIterator(JavaBinUpdateRequestCodec.java:135)
at org.apache.solr.common.util.JavaBinCodec.readVal(= JavaBinCodec.java:241)
at org.ap= ache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(Ja= vaBinUpdateRequestCodec.java:121)
= at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:206= )
at org.apache.solr.common.util= .JavaBinCodec.unmarshal(JavaBinCodec.java:126)
= at org.apache.solr.client.solrj.request.JavaBinUpdateRequest= Codec.unmarshal(JavaBinUpdateRequestCodec.java:186)
at org.apache.solr.handler.loader.JavabinLoader.parseAn= dLoadDocs(JavabinLoader.java:111)
= at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:5= 8)
at org.apache.solr.handler.Up= dateRequestHandler$1.load(UpdateRequestHandler.java:98)
at org.apache.solr.handler.ContentStreamHandlerBase= .handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handle= Request(RequestHandlerBase.java:143)
= at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064) at org.apache.solr.servlet.HttpSolrCall.e= xecute(HttpSolrCall.java:654)
at= org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450) at org.apache.solr.servlet.SolrDispatchFi= lter.doFilter(SolrDispatchFilter.java:227)
= at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispa= tchFilter.java:196)
at org.eclip= se.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:16= 52)
at org.eclipse.jetty.servlet= .ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(= ScopedHandler.java:143)
at org.e= clipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.= SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandl= e(ContextHandler.java:1127)
at o= rg.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.sessio= n.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScop= e(ContextHandler.java:1061)
at o= rg.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141= )
at org.eclipse.jetty.server.ha= ndler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handle= r.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.HandlerWrapper.h= andle(HandlerWrapper.java:97)
at= org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(Htt= pChannel.java:310)
at org.eclips= e.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnectio= n$2.run(AbstractConnection.java:540)
= at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadP= ool.java:635)
at org.eclipse.jet= ty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
= at java.lang.Thread.run(Thread.java:785)
Caused by: java.util.concurrent.TimeoutEx= ception: Idle timeout expired: 50000/50000 ms
= at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTime= out.java:161)
at org.eclipse.jet= ty.io.IdleTimeout$1.run(IdleTimeout.java:50)
= at java.util.concurrent.Executors$RunnableAdapter.call(Executo= rs.java:511)
at java.util.concur= rent.FutureTask.run(FutureTask.java:267)
= at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutur= eTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$= ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.ru= nWorker(ThreadPoolExecutor.java:1143)
= at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:618)
... 1 more =

Thanks in advance.

Regards,
Sanjay

3D"Inactive"Sanjay Lokhande" ---08/22/2017 09:00:38 PM---Hi guys, = I having 5 node solr cloud setup. The logs indicates leader
<= br>From: = "Sanjay Lokhande" <sanjaylokhande@in.ibm.com>
To: solr-use= r@lucene.apache.org
Date: = 08/22/2017 09:00 PM
Subject: Few document repli= cation not happen in solr cloud






Hi guys,
           I having = 5 node solr cloud setup.  The logs indicates leader
and 2 solr node= receiving document add request.  The other 2 nodes logs did
not sh= ow the entry to add the document and also these nodes missing the
docume= nt.
             - how such issue ca= n be troubleshoot? any pointers?
          &nb= sp; - any logging level which can help is this situation?

Regards,Sanjay





--1__=EABB0B17DFCFCF108f9e8a93df938690918cEABB0B17DFCFCF10-- --0__=EABB0B17DFCFCF108f9e8a93df938690918cEABB0B17DFCFCF10--