lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar <susheel2...@gmail.com>
Subject Re: POSSIBLE RESOURCE LEAK- solr 6.6.2
Date Sat, 22 Sep 2018 14:05:06 GMT
It looks like it got into this state of recovery after "ZooKeeper session
was expired" and then never recovered.  In what situations this warn
message "Zookeeper session was expired" can come and how does it puts into
recovery...  Any suggestions ?


2018-09-19 22:48:44.945 INFO
(searcherExecutor-7-thread-1-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.SolrCore
[COLL_shard8_replica2] Registered new searcher
Searcher@6d5ccad4[COLL_shard8_replica2]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8j4y(6.6.2):C879512/181359:delGen=2713)
Uninverting(_8z2y(6.6.2):C801846/163497:delGen=4342)
Uninverting(_81f9(6.6.2):C774110/378113:delGen=2398)
Uninverting(_82ll(6.6.2):C769263/348530:delGen=2427)
Uninverting(_83ne(6.6.2):C848884/94595:delGen=2131)
Uninverting(_4l73(6.6.2):C814472/317397:delGen=2514)
Uninverting(_7yqo(6.6.2):C724552/169735:delGen=3154)
Uninverting(_852r(6.6.2):C707686/75053:delGen=3037)
Uninverting(_ejht(6.6.2):C725012/321185:delGen=3588)
Uninverting(_gxvh(6.6.2):c432789/82230:delGen=1073)
Uninverting(_gblg(6.6.2):c215311/112981:delGen=1780)
Uninverting(_h8mr(6.6.2):c92526/13326:delGen=268)
Uninverting(_h71f(6.6.2):c51823/8823:delGen=371)
Uninverting(_h1p2(6.6.2):c58771/20717:delGen=617)
Uninverting(_h8uz(6.6.2):c9264/2802:delGen=143)
Uninverting(_hb7e(6.6.2):c35846/399:delGen=18)
Uninverting(_h9m4(6.6.2):c12335/1404:delGen=111)
Uninverting(_haiz(6.6.2):c9691/1786:delGen=67)
Uninverting(_halr(6.6.2):c960/62:delGen=24)
Uninverting(_hb16(6.6.2):c1196/69:delGen=12)
Uninverting(_hb5q(6.6.2):c2112/619:delGen=6)
Uninverting(_hb6u(6.6.2):c648/65:delGen=8)
Uninverting(_hb74(6.6.2):c664/35:delGen=9)
Uninverting(_hbba(6.6.2):c465/70:delGen=1)
Uninverting(_hbcf(6.6.2):c269/1:delGen=1)
Uninverting(_hba6(6.6.2):c504/26:delGen=3) Uninverting(_hbcq(6.6.2):c377)
Uninverting(_hbaq(6.6.2):c248/4:delGen=4)
Uninverting(_hbc4(6.6.2):c443/1:delGen=1)
Uninverting(_hb90(6.6.2):C875/376:delGen=1)
Uninverting(_hbbu(6.6.2):c431/1:delGen=1)
Uninverting(_hbbk(6.6.2):c262/2:delGen=2)
Uninverting(_hbcs(6.6.2):C9/1:delGen=1)
Uninverting(_hbct(6.6.2):C7/2:delGen=1)
Uninverting(_hbcu(6.6.2):C47/1:delGen=1)
Uninverting(_hbcv(6.6.2):C147/1:delGen=1) Uninverting(_hbcw(6.6.2):C1)
Uninverting(_hbcx(6.6.2):C4/2:delGen=1)
Uninverting(_hbcy(6.6.2):C6/3:delGen=1)))}
@
@
@
@
/recovery
2018-09-19 23:45:18.846 WARN
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@28bc45b1 name:
ZooKeeperConnection
Watcher:zk1:2182,zk2:2182,zk3:2182,zk4:2182,zk5:2182/chroot got event
WatchedEvent state:Disconnected type:None path:null path: null type: None
2018-09-19 23:45:18.846 WARN
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.c.ConnectionManager zkClient has disconnected
2018-09-19 23:45:19.185 INFO  (commitScheduler-11-thread-1) [c:COLL
s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.s.SolrIndexSearcher
Opening [Searcher@711c5774[COLL_shard8_replica2] main]
2018-09-19 23:45:19.186 INFO  (commitScheduler-11-thread-1) [c:COLL
s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.u.DirectUpdateHandler2
end_commit_flush
2018-09-19 23:45:19.496 INFO  (commitScheduler-10-thread-1) [c:COLL
s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.u.DirectUpdateHandler2
end_commit_flush
2018-09-19 23:45:19.506 WARN
(zkCallback-5-thread-50-processing-n:server62:8080_solr-SendThread(zk2:2182))
[   ] o.a.z.ClientCnxn Unable to reconnect to ZooKeeper service, session
0x36591d677ff0003 has expired
2018-09-19 23:45:19.507 WARN
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@28bc45b1 name:
ZooKeeperConnection
Watcher:zk1:2182,zk2:2182,zk3:2182,zk4:2182,zk5:2182/chroot got event
WatchedEvent state:Expired type:None path:null path: null type: None
2018-09-19 23:45:19.507 WARN
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.c.ConnectionManager Our previous ZooKeeper session was expired.
Attempting to reconnect to recover relationship with ZooKeeper...
2018-09-19 23:45:19.507 WARN
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.c.DefaultConnectionStrategy Connection expired - starting a new
one...
2018-09-19 23:45:19.518 INFO
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.c.ConnectionManager Connection with ZooKeeper reestablished.
2018-09-19 23:45:19.518 INFO
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing core
states after session expiration.
2018-09-19 23:45:19.521 INFO
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (16) -> (15)
2018-09-19 23:45:19.542 INFO
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.ZkController Replica core_node16 NOT in leader-initiated recovery,
need to wait for leader to see down state.
2018-09-19 23:45:19.605 INFO
(zkCallback-5-thread-89-processing-n:server62:8080_solr) [   ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/COLLP/state.json] for collection [COLLP] has occurred -
updating... (live nodes size: [15])
2018-09-19 23:45:19.754 INFO
(zkCallback-5-thread-89-processing-n:server62:8080_solr) [   ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged path:/collections/COLL/state.json]
for collection [COLL] has occurred - updating... (live nodes size: [15])
2018-09-19 23:45:20.741 INFO
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.ZkController Replica core_node16 NOT in leader-initiated recovery,
need to wait for leader to see down state.
2018-09-19 23:45:20.743 INFO
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.ZkController Register node as live in
ZooKeeper:/live_nodes/server62:8080_solr
2018-09-19 23:45:20.747 INFO
(zkCallback-5-thread-89-processing-n:server62:8080_solr) [   ]
o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (15) -> (16)
2018-09-19 23:45:20.747 INFO
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.c.DefaultConnectionStrategy Reconnected to ZooKeeper
2018-09-19 23:45:20.747 INFO
(coreZkRegister-1-thread-6-processing-n:server62:8080_solr) [   ]
o.a.s.c.ZkController Registering core COLL_shard8_replica2 afterExpiration?
true
2018-09-19 23:45:20.748 INFO
(zkCallback-5-thread-50-processing-n:server62:8080_solr-EventThread) [   ]
o.a.s.c.c.ConnectionManager Connected:true
2018-09-19 23:45:20.748 INFO
(coreZkRegister-1-thread-7-processing-n:server62:8080_solr) [   ]
o.a.s.c.ZkController Registering core COLLP_shard8_replica2
afterExpiration? true
2018-09-19 23:45:20.749 INFO  (Thread-57617) [   ] o.a.s.c.SolrCore config
update listener called for core COLL_shard8_replica2
2018-09-19 23:45:20.751 INFO  (Thread-57618) [   ] o.a.s.c.SolrCore config
update listener called for core COLLP_shard8_replica2
2018-09-19 23:45:20.753 INFO  (Thread-57618) [   ] o.a.s.c.SolrCore core
reload COLLP_shard8_replica2
2018-09-19 23:45:20.753 INFO
(coreZkRegister-1-thread-6-processing-n:server62:8080_solr) [c:COLL
s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.ZkController Core
needs to recover:COLL_shard8_replica2
2018-09-19 23:45:20.753 INFO
(coreZkRegister-1-thread-7-processing-n:server62:8080_solr) [c:COLLP
s:shard8 r:core_node16 x:COLLP_shard8_replica2] o.a.s.c.ZkController Core
needs to recover:COLLP_shard8_replica2
2018-09-19 23:45:20.753 INFO
(updateExecutor-2-thread-5-processing-n:server62:8080_solr
x:COLLP_shard8_replica2 s:shard8 c:COLLP r:core_node16) [c:COLLP s:shard8
r:core_node16 x:COLLP_shard8_replica2] o.a.s.u.DefaultSolrCoreState Running
recovery
2018-09-19 23:45:20.753 INFO
(updateExecutor-2-thread-5-processing-n:server62:8080_solr
x:COLLP_shard8_replica2 s:shard8 c:COLLP r:core_node16) [c:COLLP s:shard8
r:core_node16 x:COLLP_shard8_replica2] o.a.s.c.ActionThrottle The last
recovery attempt started 1026739687ms ago.
2018-09-19 23:45:20.764 INFO  (Thread-57618) [   ] o.a.s.c.RequestParams
conf resource params.json loaded . version : 0
2018-09-19 23:45:20.764 INFO  (Thread-57618) [   ] o.a.s.c.RequestParams
request params refreshed to version 0
2018-09-19 23:45:20.765 INFO  (Thread-57618) [   ]
o.a.s.c.SolrResourceLoader [COLLP_shard8_replica2] Added 55 libs to
classloader, from paths: [/opt/solr/contrib/clustering/lib,
/opt/solr/contrib/extraction/lib, /opt/solr/contrib/langid/lib,
/opt/solr/contrib/velocity/lib, /opt/solr/dist]
2018-09-19 23:45:20.789 INFO
(recoveryExecutor-3-thread-5-processing-n:server62:8080_solr
x:COLLP_shard8_replica2 s:shard8 c:COLLP r:core_node16) [c:COLLP s:shard8
r:core_node16 x:COLLP_shard8_replica2] o.a.s.c.RecoveryStrategy Starting
recovery process. recoveringAfterStartup=false
2018-09-19 23:45:20.792 INFO
(updateExecutor-2-thread-6-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.u.DefaultSolrCoreState Running
recovery
2018-09-19 23:45:20.792 INFO
(updateExecutor-2-thread-6-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.ActionThrottle The last
recovery attempt started 1398949248ms ago.
2018-09-19 23:45:20.792 INFO
(recoveryExecutor-3-thread-6-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Starting
recovery process. recoveringAfterStartup=false
2018-09-19 23:45:20.792 INFO
(recoveryExecutor-3-thread-5-processing-n:server62:8080_solr
x:COLLP_shard8_replica2 s:shard8 c:COLLP r:core_node16) [c:COLLP s:shard8
r:core_node16 x:COLLP_shard8_replica2] o.a.s.c.RecoveryStrategy Begin
buffering updates. core=[COLLP_shard8_replica2]
2018-09-19 23:45:20.792 INFO
(recoveryExecutor-3-thread-5-processing-n:server62:8080_solr
x:COLLP_shard8_replica2 s:shard8 c:COLLP r:core_node16) [c:COLLP s:shard8
r:core_node16 x:COLLP_shard8_replica2] o.a.s.u.UpdateLog Starting to buffer
updates. FSUpdateLog{state=ACTIVE, tlog=null}
2018-09-19 23:45:20.792 INFO
(recoveryExecutor-3-thread-5-processing-n:server62:8080_solr
x:COLLP_shard8_replica2 s:shard8 c:COLLP r:core_node16) [c:COLLP s:shard8
r:core_node16 x:COLLP_shard8_replica2] o.a.s.c.RecoveryStrategy Publishing
state of core [COLLP_shard8_replica2] as recovering, leader is [
http://server61:8080/solr/COLLP_shard8_replica1/] and I am [
http://server62:8080/solr/COLLP_shard8_replica2/]
2018-09-19 23:45:20.832 INFO
(recoveryExecutor-3-thread-5-processing-n:server62:8080_solr
x:COLLP_shard8_replica2 s:shard8 c:COLLP r:core_node16) [c:COLLP s:shard8
r:core_node16 x:COLLP_shard8_replica2] o.a.s.c.RecoveryStrategy Sending
prep recovery command to [http://server61:8080/solr]; [WaitForState:
action=PREPRECOVERY&core=COLLP_shard8_replica1&nodeName=server62:8080_solr&coreNodeName=core_node16&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
2018-09-19 23:45:20.861 INFO  (Thread-57618) [   ] o.a.s.u.SolrIndexConfig
IndexWriter infoStream solr logging is enabled
2018-09-19 23:45:20.863 INFO  (Thread-57618) [   ] o.a.s.c.SolrConfig Using
Lucene MatchVersion: 6.6.2
2018-09-19 23:45:20.924 INFO  (Thread-57618) [   ] o.a.s.s.IndexSchema
[COLLP_shard8_replica2] Schema name=COLLP
2018-09-19 23:45:20.931 INFO
(zkCallback-5-thread-89-processing-n:server62:8080_solr) [   ]
o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/COLLP/state.json] for collection [COLLP] has occurred -
updating... (live nodes size: [16])
2018-09-19 23:45:21.075 INFO
(recoveryExecutor-3-thread-6-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Begin
buffering updates. core=[COLL_shard8_replica2]
2018-09-19 23:45:21.075 INFO
(recoveryExecutor-3-thread-6-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.u.UpdateLog Starting to buffer
updates. FSUpdateLog{state=ACTIVE,
tlog=tlog{file=/app/solr/data/COLL_shard8_replica2/data/tlog/tlog.0000000000000051065.1612081196978667520
refcount=1}}
2018-09-19 23:45:21.075 INFO
(recoveryExecutor-3-thread-6-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Publishing
state of core [COLL_shard8_replica2] as recovering, leader is [
http://server61:8080/solr/COLL_shard8_replica1/] and I am [
http://server62:8080/solr/COLL_shard8_replica2/]
2018-09-19 23:45:21.079 INFO
(recoveryExecutor-3-thread-6-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Sending prep
recovery command to [http://server61:8080/solr]; [WaitForState:
action=PREPRECOVERY&core=COLL_shard8_replica1&nodeName=server62:8080_solr&coreNodeName=core_node16&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
2018-09-19 23:45:21.084 INFO
(searcherExecutor-7-thread-1-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.QuerySenderListener
QuerySenderListener sending requests to Searcher@711c5774[COLL_shard8_replica2]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8j4y(6.6.2):C879512/181368:delGen=2714)
Uninverting(_8z2y(6.6.2):C801846/163505:delGen=4343)
Uninverting(_81f9(6.6.2):C774110/378113:delGen=2398)
Uninverting(_82ll(6.6.2):C769263/348530:delGen=2427)
Uninverting(_83ne(6.6.2):C848884/94597:delGen=2133)
Uninverting(_4l73(6.6.2):C814472/317397:delGen=2514)
Uninverting(_7yqo(6.6.2):C724552/169736:delGen=3155)
Uninverting(_852r(6.6.2):C707686/75054:delGen=3038)
Uninverting(_ejht(6.6.2):C725012/321262:delGen=3591)
Uninverting(_gxvh(6.6.2):c432789/82761:delGen=1078)
Uninverting(_gblg(6.6.2):c215311/113006:delGen=1782)
Uninverting(_h8mr(6.6.2):c92526/14080:delGen=277)
Uninverting(_h71f(6.6.2):c51823/8823:delGen=371)
Uninverting(_h1p2(6.6.2):c58771/20719:delGen=618)
Uninverting(_h8uz(6.6.2):c9264/2802:delGen=143)
Uninverting(_hb7e(6.6.2):c35846/429:delGen=27)
Uninverting(_h9m4(6.6.2):c12335/1407:delGen=114)
Uninverting(_haiz(6.6.2):c9691/1789:delGen=68)
Uninverting(_halr(6.6.2):c960/340:delGen=25)
Uninverting(_hb16(6.6.2):c1196/69:delGen=12)
Uninverting(_hb5q(6.6.2):c2112/867:delGen=7)
Uninverting(_hb6u(6.6.2):c648/65:delGen=8)
Uninverting(_hb74(6.6.2):c664/36:delGen=10)
Uninverting(_hbba(6.6.2):c465/72:delGen=3)
Uninverting(_hbd1(6.6.2):c1224/1:delGen=1)
Uninverting(_hba6(6.6.2):c504/31:delGen=6)
Uninverting(_hbcq(6.6.2):c377/1:delGen=1)
Uninverting(_hbe7(6.6.2):c2163/1:delGen=1)
Uninverting(_hbdx(6.6.2):c506/3:delGen=1)
Uninverting(_hbfl(6.6.2):c71/1:delGen=1)
Uninverting(_hbfb(6.6.2):c62/1:delGen=1)
Uninverting(_hbed(6.6.2):C206/3:delGen=1)
Uninverting(_hbfm(6.6.2):C6/2:delGen=1)
Uninverting(_hbfn(6.6.2):C560/281:delGen=2)
Uninverting(_hbfo(6.6.2):C12/7:delGen=1)
Uninverting(_hbfp(6.6.2):C4/2:delGen=1)
Uninverting(_hbfq(6.6.2):C4/2:delGen=1)
Uninverting(_hbfs(6.6.2):C9/5:delGen=1)))}
2018-09-19 23:45:21.084 INFO
(searcherExecutor-7-thread-1-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.QuerySenderListener
QuerySenderListener done.
2018-09-19 23:45:21.084 INFO
(searcherExecutor-7-thread-1-processing-n:server62:8080_solr
x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8
r:core_node16 x:COLL_shard8_replica2] o.a.s.c.SolrCore
[COLL_shard8_replica2] Registered new searcher
Searcher@711c5774[COLL_shard8_replica2]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_8j4y(6.6.2):C879512/181368:delGen=2714)
Uninverting(_8z2y(6.6.2):C801846/163505:delGen=4343)
Uninverting(_81f9(6.6.2):C774110/378113:delGen=2398)
Uninverting(_82ll(6.6.2):C769263/348530:delGen=2427)
Uninverting(_83ne(6.6.2):C848884/94597:delGen=2133)
Uninverting(_4l73(6.6.2):C814472/317397:delGen=2514)
Uninverting(_7yqo(6.6.2):C724552/169736:delGen=3155)
Uninverting(_852r(6.6.2):C707686/75054:delGen=3038)
Uninverting(_ejht(6.6.2):C725012/321262:delGen=3591)
Uninverting(_gxvh(6.6.2):c432789/82761:delGen=1078)
Uninverting(_gblg(6.6.2):c215311/113006:delGen=1782)
Uninverting(_h8mr(6.6.2):c92526/14080:delGen=277)
Uninverting(_h71f(6.6.2):c51823/8823:delGen=371)
Uninverting(_h1p2(6.6.2):c58771/20719:delGen=618)
Uninverting(_h8uz(6.6.2):c9264/2802:delGen=143)
Uninverting(_hb7e(6.6.2):c35846/429:delGen=27)
Uninverting(_h9m4(6.6.2):c12335/1407:delGen=114)
Uninverting(_haiz(6.6.2):c9691/1789:delGen=68)
Uninverting(_halr(6.6.2):c960/340:delGen=25)
Uninverting(_hb16(6.6.2):c1196/69:delGen=12)
Uninverting(_hb5q(6.6.2):c2112/867:delGen=7)
Uninverting(_hb6u(6.6.2):c648/65:delGen=8)
Uninverting(_hb74(6.6.2):c664/36:delGen=10)
Uninverting(_hbba(6.6.2):c465/72:delGen=3)
Uninverting(_hbd1(6.6.2):c1224/1:delGen=1)
Uninverting(_hba6(6.6.2):c504/31:delGen=6)
Uninverting(_hbcq(6.6.2):c377/1:delGen=1)
Uninverting(_hbe7(6.6.2):c2163/1:delGen=1)
Uninverting(_hbdx(6.6.2):c506/3:delGen=1)
Uninverting(_hbfl(6.6.2):c71/1:delGen=1)
Uninverting(_hbfb(6.6.2):c62/1:delGen=1)
Uninverting(_hbed(6.6.2):C206/3:delGen=1)
Uninverting(_hbfm(6.6.2):C6/2:delGen=1)
Uninverting(_hbfn(6.6.2):C560/281:delGen=2)
Uninverting(_hbfo(6.6.2):C12/7:delGen=1)
Uninverting(_hbfp(6.6.2):C4/2:delGen=1)
Uninverting(_hbfq(6.6.2):C4/2:delGen=1)
Uninverting(_hbfs(6.6.2):C9/5:delGen=1)))}

On Sat, Sep 22, 2018 at 2:20 AM Susheel Kumar <susheel2777@gmail.com> wrote:

> Hello,
>
> I noticed one of the replica's in Recover Failed status and after trying
> to recreate the replica/restart the node to recover, I see below error from
> solr log.  The leader for this shard8 replica2 i.e. replica1@server 61
> seems to be fine and serving the queries.
>
> What does this indicate "SolrIndexWriter was not closed prior to
> finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!" from below log.
>
> 2018-09-22 06:00:13.686 ERROR (Finalizer) [   ] o.a.s.u.SolrIndexWriter Error closing IndexWriter2018-09-22 05:58:13.409 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=5
> 2018-09-22 05:58:13.686 INFO  (qtp834133664-19) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=39
> 2018-09-22 05:58:19.338 ERROR (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: Index fetch failed :
>         at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:598)
>         at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:301)
>         at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:400)
>         at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:219)
>         at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:471)
>         at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:284)
>         at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1997958933 vs expected header=1071082519 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica2/data/index/_8ktm.fnm")))
>         at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196)
>         at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255)
>         at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:117)
>         at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1063)
>         at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1075)
>         at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:960)
>         at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:118)
>         at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:93)
>         at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
>         at org.apache.solr.update.DefaultSolrCoreState.changeWriter(DefaultSolrCoreState.java:220)
>         at org.apache.solr.update.DefaultSolrCoreState.openIndexWriter(DefaultSolrCoreState.java:245)
>         at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:551)
>         ... 12 more
>         Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=e42fdf3e actual=c0432e62 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica2/data/index/_8ktm.fnm")))
>                 at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
>                 at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
>                 at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:171)
>                 ... 21 more
>
> 2018-09-22 05:58:19.338 ERROR (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed.
>         at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:222)
>         at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:471)
>         at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:284)
>         at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
> 2018-09-22 05:58:19.338 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still buffering updates.
> 2018-09-22 05:58:19.338 ERROR (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Recovery failed - trying again... (0)
> 2018-09-22 05:58:19.338 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Wait [2.0] seconds before trying to recover again (attempt=1)
> 2018-09-22 05:58:23.352 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=0
> 2018-09-22 05:58:23.486 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=18
> 2018-09-22 05:58:23.763 INFO  (qtp834133664-22) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=53
> 2018-09-22 05:58:29.339 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[COLL_shard8_replica2]
> 2018-09-22 05:58:29.339 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.u.UpdateLog Restarting buffering. previous=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=0}
> 2018-09-22 05:58:29.339 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=BUFFERING, tlog=tlog{file=/app/solr/data/COLL_shard8_replica2/data/tlog/tlog.0000000000000000000.1612285920588857344 refcount=2}}
> 2018-09-22 05:58:29.339 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Publishing state of core [COLL_shard8_replica2] as recovering, leader is [http://server61:8080/solr/COLL_shard8_replica1/] and I am [http://server62:8080/solr/COLL_shard8_replica2/]
> 2018-09-22 05:58:29.343 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://server61:8080/solr]; [WaitForState: action=PREPRECOVERY&core=COLL_shard8_replica1&nodeName=server62:8080_solr&coreNodeName=core_node16&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
> 2018-09-22 05:58:29.351 INFO  (zkCallback-5-thread-23-processing-n:server62:8080_solr) [   ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/COLL/state.json] for collection [COLL] has occurred - updating... (live nodes size: [16])
> 2018-09-22 05:58:31.846 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Starting Replication Recovery.
> 2018-09-22 05:58:31.846 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Attempting to replicate from [http://server61:8080/solr/COLL_shard8_replica1/].
> 2018-09-22 05:58:32.234 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Master's generation: 51970
> 2018-09-22 05:58:32.234 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Master's version: 1537595911854
> 2018-09-22 05:58:32.234 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Slave's generation: 1
> 2018-09-22 05:58:32.234 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Slave's version: 0
> 2018-09-22 05:58:32.234 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Starting replication process
> 2018-09-22 05:58:32.290 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Number of files in latest index in master: 257
> 2018-09-22 05:58:32.309 ERROR (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: Index fetch failed :
>         at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:598)
>         at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:301)
>         at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:400)
>         at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:219)
>         at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:471)
>         at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:284)
>         at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1997958933 vs expected header=1071082519 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica2/data/index/_8ktm.fnm")))
>         at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196)
>         at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255)
>         at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:117)
>         at org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1063)
>         at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1075)
>         at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:960)
>         at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:118)
>         at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:93)
>         at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:257)
>         at org.apache.solr.update.DefaultSolrCoreState.changeWriter(DefaultSolrCoreState.java:220)
>         at org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:229)
>         at org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(DirectUpdateHandler2.java:726)
>         at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:457)
>         ... 12 more
>         Suppressed: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=e42fdf3e actual=c0432e62 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard8_replica2/data/index/_8ktm.fnm")))
>                 at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
>                 at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:462)
>                 at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:171)
>                 ... 22 more
>
> 2018-09-22 05:58:32.309 ERROR (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed.
>         at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:222)
>         at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:471)
>         at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:284)
>         at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
> 2018-09-22 05:58:32.310 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still buffering updates.
> 2018-09-22 05:58:32.310 ERROR (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1)
> 2018-09-22 05:58:32.310 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Wait [4.0] seconds before trying to recover again (attempt=2)
> 2018-09-22 05:58:33.423 INFO  (qtp834133664-15) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=4
> 2018-09-22 05:58:33.548 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=3
> 2018-09-22 05:58:33.834 INFO  (qtp834133664-19) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=44
> 2018-09-22 05:58:43.480 INFO  (qtp834133664-13) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=2
> 2018-09-22 05:58:43.624 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=8
> 2018-09-22 05:58:43.882 INFO  (qtp834133664-22) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=23
> 2018-09-22 05:58:52.310 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[COLL_shard8_replica2]
> 2018-09-22 05:58:52.310 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.u.UpdateLog Restarting buffering. previous=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=2630296}
> 2018-09-22 05:58:52.310 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=BUFFERING, tlog=tlog{file=/app/solr/data/COLL_shard8_replica2/data/tlog/tlog.0000000000000000000.1612285920588857344 refcount=2}}
> 2018-09-22 05:58:52.310 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Publishing state of core [COLL_shard8_replica2] as recovering, leader is [http://server61:8080/solr/COLL_shard8_replica1/] and I am [http://server62:8080/solr/COLL_shard8_replica2/]
> 2018-09-22 05:58:52.317 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://server61:8080/solr]; [WaitForState: action=PREPRECOVERY&core=COLL_shard8_replica1&nodeName=server62:8080_solr&coreNodeName=core_node16&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
> 2018-09-22 05:58:52.325 INFO  (zkCallback-5-thread-23-processing-n:server62:8080_solr) [   ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/COLL/state.json] for collection [COLL] has occurred - updating... (live nodes size: [16])
> 2018-09-22 05:58:53.538 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=8
> 2018-09-22 05:58:53.679 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=4
> 2018-09-22 05:58:53.983 INFO  (qtp834133664-19) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=63
> 2018-09-22 05:58:54.820 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Starting Replication Recovery.
> 2018-09-22 05:58:54.820 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.RecoveryStrategy Attempting to replicate from [http://server61:8080/solr/COLL_shard8_replica1/].
> 2018-09-22 05:58:54.967 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Master's generation: 51971
> 2018-09-22 05:58:54.967 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Master's version: 1537595934827
> 2018-09-22 05:58:54.967 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Slave's generation: 1
> 2018-09-22 05:58:54.967 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Slave's version: 0
> 2018-09-22 05:58:54.967 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Starting replication process
> 2018-09-22 05:58:55.018 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Number of files in latest index in master: 173
> 2018-09-22 05:58:55.019 INFO  (recoveryExecutor-3-thread-3-processing-n:server62:8080_solr x:COLL_shard8_replica2 s:shard8 c:COLL r:core_node16) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.h.IndexFetcher Starting download (fullCopy=true) to NRTCachingDirectory(MMapDirectory@/app/solr/data/COLL_shard8_replica2/data/index.20180922055855018 lockFactory=org.apache.lucene.store.NativeFSLockFactory@74255cc4; maxCacheMB=48.0 maxMergeSizeMB=4.0)
> 2018-09-22 05:59:03.590 INFO  (qtp834133664-15) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=0
> 2018-09-22 05:59:03.781 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=25
> 2018-09-22 05:59:04.061 INFO  (qtp834133664-22) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=52
> 2018-09-22 05:59:13.125 INFO  (qtp834133664-13) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=42
> 2018-09-22 05:59:13.672 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=2
> 2018-09-22 05:59:13.812 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=9
> 2018-09-22 05:59:23.229 INFO  (qtp834133664-19) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=43
> 2018-09-22 05:59:23.275 INFO  (qtp834133664-15) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=18
> 2018-09-22 05:59:23.732 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=3
> 2018-09-22 05:59:23.915 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=18
> 2018-09-22 05:59:33.390 INFO  (qtp834133664-22) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=89
> 2018-09-22 05:59:33.778 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=0
> 2018-09-22 05:59:33.958 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=19
> 2018-09-22 05:59:43.461 INFO  (qtp834133664-13) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=46
> 2018-09-22 05:59:43.862 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=2
> 2018-09-22 05:59:44.009 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=15
> 2018-09-22 05:59:53.515 INFO  (qtp834133664-19) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=24
> 2018-09-22 05:59:53.915 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=4
> 2018-09-22 05:59:54.075 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=15
> 2018-09-22 06:00:03.177 INFO  (qtp834133664-15) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=21
> 2018-09-22 06:00:03.596 INFO  (qtp834133664-22) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=51
> 2018-09-22 06:00:03.977 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=12
> 2018-09-22 06:00:13.207 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=4
> 2018-09-22 06:00:13.685 INFO  (qtp834133664-13) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=65
> 2018-09-22 06:00:13.686 ERROR (Finalizer) [   ] o.a.s.u.SolrIndexWriter SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
> 2018-09-22 06:00:13.686 ERROR (Finalizer) [   ] o.a.s.u.SolrIndexWriter Error closing IndexWriter
> java.lang.NullPointerException
>         at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3414)
>         at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3376)
>         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1126)
>         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1171)
>         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:278)
>         at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:334)
>         at java.lang.System$2.invokeFinalize(System.java:1270)
>         at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
>         at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
> 2018-09-22 06:00:13.686 ERROR (Finalizer) [   ] o.a.s.u.SolrIndexWriter SolrIndexWriter was not closed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
> 2018-09-22 06:00:13.687 ERROR (Finalizer) [   ] o.a.s.u.SolrIndexWriter Error closing IndexWriter
> java.lang.NullPointerException
>         at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3414)
>         at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3376)
>         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:1126)
>         at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1171)
>         at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:278)
>         at org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:334)
>         at java.lang.System$2.invokeFinalize(System.java:1270)
>         at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98)
>         at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210)
> 2018-09-22 06:00:14.018 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=7
> 2018-09-22 06:00:23.270 INFO  (qtp834133664-15) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=15
> 2018-09-22 06:00:23.775 INFO  (qtp834133664-19) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=42
> 2018-09-22 06:00:24.058 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=0
> 2018-09-22 06:00:33.139 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=19
> 2018-09-22 06:00:33.345 INFO  (qtp834133664-15) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=24
> 2018-09-22 06:00:33.844 INFO  (qtp834133664-22) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=43
> 2018-09-22 06:00:43.165 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=0
> 2018-09-22 06:00:43.398 INFO  (qtp834133664-15) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=8
> 2018-09-22 06:00:43.915 INFO  (qtp834133664-13) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=46
> 2018-09-22 06:00:53.211 INFO  (qtp834133664-21) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=2
> 2018-09-22 06:00:53.448 INFO  (qtp834133664-15) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=0
> 2018-09-22 06:00:53.987 INFO  (qtp834133664-19) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=49
> 2018-09-22 06:01:03.252 INFO  (qtp834133664-20) [c:COLL s:shard8 r:core_node16 x:COLL_shard8_replica2] o.a.s.c.S.Request [COLL_shard8_replica2]  webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&distrib.from=http://server61:8080/solr/COLL_shard8_replica1/&wt=javabin&version=2} status=0 QTime=0
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message