lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vannia Rajan <van...@knackforge.com>
Subject Re: SPLITSHARD in SOLR 5.5.1
Date Fri, 25 Aug 2017 12:50:11 GMT
Hi,

I figured out when this issue happens, by observing the patterns with a
small set of data.

SPLITSHARD issues a soft-commit (with some of the files still not fully
written to disk). If I restart SOLR without issuing an explicit <commit />,
the index directory is not fully written and the process is killed. During
next restart, the incomplete index is set to 0 records and cleaned up.

I think we should update the documentations to let users know that they
need to issue a hard <commit /> immediately after a SPLITSHARD operation. I
will mention this in the JIRA case I opened for this -
https://issues.apache.org/jira/browse/SOLR-11287.

Thanks for the help!

Thanks,
Vanniarajan

On Fri, Aug 25, 2017 at 5:29 AM, Erick Erickson <erickerickson@gmail.com>
wrote:

> This is strange. You can use the "bin/solr zk cp..." commands to bring
> the collection state.json down to your local machine and do anything
> you want to do with it, including marking the old shard as "active"
> and removing the entries for the sub-shards. It's just a text file
> after all. I'd also delete the sub-shard directories from my hard
> disks.
>
> Do this _very_ cautiously! And I'd stop all my Solr nodes before doing
> anything with it. And copy the original index somewhere for
> safekeeping.
>
> So, let's assume this gets your data back and you try the SPLITSHARD
> again. Once that's complete, I'd check the directories of the
> sub-shards. Are they (roughly) half the size of your old shard? If so,
> copy them somewhere while you investigate what is deleting the data
> directories, Solr should absolutely not be doing that. My guess is
> it's something external to Solr that wiped them somehow, but how I
> have no clue.
>
> Best,
> Erick
>
> On Thu, Aug 24, 2017 at 1:23 PM, Vannia Rajan <vannia@knackforge.com>
> wrote:
> > Binoy,
> >
> > I don't see anything wrong with the logs. The newly split shards are up,
> > with the parent set to inactive. But before restart, the new shards had
> all
> > of the data as in parent. After a restart, it wiped off to 0. I checked
> the
> > data directory, the data is completely gone.
> >
> > Below, I attach the log messages that happened during the splitshard
> > operation completion & when the next restart of SOLR was done.
> >
> > Collection = COLLECTION
> > Shard that lost data = shard2 (xxx.xx.xx.xxx:8983_solr)
> >
> > *Log messages at the end of splitshard operation:*
> >
> >> INFO  - 2017-08-23 21:58:25.642; [   ]
> >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Index on shard:
> >> xxx.xx.xx.xxx:8983_solr split into two successfully
> >> INFO  - 2017-08-23 21:58:25.642; [   ]
> >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Applying
> buffered
> >> updates on : COLLECTION_shard2_0_replica1
> >> INFO  - 2017-08-23 21:58:25.643; [   ]
> >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Applying
> buffered
> >> updates on : COLLECTION_shard2_1_replica1
> >> INFO  - 2017-08-23 21:58:25.731; [   ]
> >> org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null
> path=/admin/cores
> >> params={async=1000487066830891264&qt=/admin/
> cores&name=COLLECTION_shard2_0_replica1&action=REQUESTAPPLYUPDATES&wt=
> javabin&version=2}
> >> status=0 QTime=43
> >> INFO  - 2017-08-23 21:58:25.774; [   ]
> >> org.apache.solr.handler.admin.CoreAdminOperation$13; Applying buffered
> >> updates on core: COLLECTION_shard2_0_replica1
> >> INFO  - 2017-08-23 21:58:25.774; [   ]
> >> org.apache.solr.handler.admin.CoreAdminOperation$13; No buffered
> updates
> >> available. core=COLLECTION_shard2_0_replica1
> >> INFO  - 2017-08-23 21:58:25.906; [   ]
> >> org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null
> path=/admin/cores
> >> params={async=1000487066831162147&qt=/admin/
> cores&name=COLLECTION_shard2_1_replica1&action=REQUESTAPPLYUPDATES&wt=
> javabin&version=2}
> >> status=0 QTime=131
> >> INFO  - 2017-08-23 21:58:25.952; [   ]
> >> org.apache.solr.handler.admin.CoreAdminOperation$13; Applying buffered
> >> updates on core: COLLECTION_shard2_1_replica1
> >> INFO  - 2017-08-23 21:58:25.952; [   ]
> >> org.apache.solr.handler.admin.CoreAdminOperation$13; No buffered
> updates
> >> available. core=COLLECTION_shard2_1_replica1
> >> INFO  - 2017-08-23 21:58:25.999; [   ]
> >> org.apache.solr.handler.admin.CoreAdminOperation$14; Checking request
> >> status for : 1000487066831162147
> >> INFO  - 2017-08-23 21:58:26.000; [   ]
> >> org.apache.solr.servlet.HttpSolrCall; [admin] webapp=null
> path=/admin/cores
> >> params={qt=/admin/cores&requestid=1000487066831162147&
> action=REQUESTSTATUS&wt=javabin&version=2}
> >> status=0 QTime=0
> >> INFO  - 2017-08-23 21:58:26.001; [   ]
> >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Successfully
> >> applied buffered updates on : [COLLECTION_shard2_0_replica1,
> >> COLLECTION_shard2_1_replica1]
> >> INFO  - 2017-08-23 21:58:26.002; [   ]
> >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Replication
> factor
> >> is 1 so switching shard states
> >> INFO  - 2017-08-23 21:58:26.004; [   ]
> >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Successfully
> >> created all replica shards for all sub-slices [shard2_0, shard2_1]
> >> INFO  - 2017-08-23 21:58:26.004; [   ]
> >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Calling soft
> commit
> >> to make sub shard updates visible
> >> INFO  - 2017-08-23 21:58:26.179; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 1, message = {
> >>   "shard2":"inactive",
> >>   "collection":"COLLECTION",
> >>   "shard2_1":"active",
> >>   "operation":"updateshardstate",
> >>   "shard2_0":"active"} current state version: 0
> >> INFO  - 2017-08-23 21:58:26.180; [   ]
> >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state
> invoked for
> >> collection: COLLECTION with message: {
> >>   "shard2":"inactive",
> >>   "collection":"COLLECTION",
> >>   "shard2_1":"active",
> >>   "operation":"updateshardstate",
> >>   "shard2_0":"active"}
> >> INFO  - 2017-08-23 21:58:26.180; [   ]
> >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard2
> to
> >> inactive
> >> INFO  - 2017-08-23 21:58:26.180; [   ]
> >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state
> shard2_1 to
> >> active
> >> INFO  - 2017-08-23 21:58:26.180; [   ]
> >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state
> shard2_0 to
> >> active
> >> INFO  - 2017-08-23 21:58:26.226; [   ]
> >> org.apache.solr.cloud.overseer.ZkStateWriter; going to
> update_collection
> >> /collections/COLLECTION/state.json version: 2689
> >> INFO  - 2017-08-23 21:58:26.271; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster
> state
> >> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> >> path:/collections/COLLECTION/state.json] for collection [COLLECTION]
> has
> >> occurred - updating... (live nodes size: [8])
> >> INFO  - 2017-08-23 21:58:26.271; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader; Updating data for
> [COLLECTION]
> >> from [2689] to [2690]
> >> INFO  - 2017-08-23 21:58:27.076; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1] org.apache.solr.update.
> DirectUpdateHandler2;
> >> start
> >> commit{,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=true,prepareCommit=false}
> >> INFO  - 2017-08-23 21:58:27.166; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.update.DirectUpdateHandler2; start
> >> commit{,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=true,prepareCommit=false}
> >> INFO  - 2017-08-23 21:58:27.255; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrCore;
> >> SolrIndexSearcher has not changed - not re-opening:
> >> org.apache.solr.search.SolrIndexSearcher
> >> INFO  - 2017-08-23 21:58:27.255; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.update.DirectUpdateHandler2; start
> >> commit{,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=true,prepareCommit=false}
> >> INFO  - 2017-08-23 21:58:27.298; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1] org.apache.solr.update.
> DirectUpdateHandler2;
> >> end_commit_flush
> >> INFO  - 2017-08-23 21:58:27.384; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1]
> >> org.apache.solr.update.processor.LogUpdateProcessorFactory$
> LogUpdateProcessor;
> >> [COLLECTION_shard2_replica1]  webapp=/solr path=/update
> >> params={update.distrib=FROMLEADER&update.chain=add-
> unknown-fields-to-the-schema&waitSearcher=true&
> openSearcher=true&commit=true&softCommit=true&distrib.from=
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_
> replica1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=}
> >> 0 354
> >> INFO  - 2017-08-23 21:58:28.754; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.search.
> SolrIndexSearcher;
> >> Opening Searcher@a4f0979[COLLECTION_shard2_1_replica1] main
> >> INFO  - 2017-08-23 21:58:28.757; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
> >> INFO  - 2017-08-23 21:58:28.758; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener sending requests to Searcher@a4f0979[COLLECTION_
> shard2_1_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader(
> Uninverting(_0(5.5.1):C1332771)
> >> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638)
> >> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033)
> >> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611)
> >> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083)
> >> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563)
> >> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123)
> >> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893)
> >> Uninverting(_s(5.5.1):c1969016) Uninverting(_t(5.5.1):c1918535)
> >> Uninverting(_u(5.5.1):c1924194) Uninverting(_w(5.5.1):c1660229)
> >> Uninverting(_x(5.5.1):c49355) Uninverting(_y(5.5.1):c150056)
> >> Uninverting(_z(5.5.1):c63319) Uninverting(_10(5.5.1):c80520)
> >> Uninverting(_11(5.5.1):c92203) Uninverting(_12(5.5.1):c96290)
> >> Uninverting(_13(5.5.1):c24621) Uninverting(_14(5.5.1):c94896)
> >> Uninverting(_15(5.5.1):c8987) Uninverting(_16(5.5.1):c12045)
> >> Uninverting(_17(5.5.1):c10943) Uninverting(_18(5.5.1):c7357)
> >> Uninverting(_19(5.5.1):c7519) Uninverting(_1a(5.5.1):c2426)
> >> Uninverting(_1b(5.5.1):c2700) Uninverting(_1c(5.5.1):c3336)
> >> Uninverting(_1d(5.5.1):c5069) Uninverting(_1e(5.5.1):c3707)
> >> Uninverting(_1f(5.5.1):c2391) Uninverting(_1g(5.5.1):c1389)
> >> Uninverting(_1h(5.5.1):c37) Uninverting(_1i(5.5.1):c42)
> >> Uninverting(_1j(5.5.1):c52)))}
> >> INFO  - 2017-08-23 21:58:28.804; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener done.
> >> INFO  - 2017-08-23 21:58:28.804; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.SolrCore;
> >> [COLLECTION_shard2_1_replica1] Registered new searcher Searcher@a4f0979
> [COLLECTION_shard2_1_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader(
> Uninverting(_0(5.5.1):C1332771)
> >> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638)
> >> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033)
> >> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611)
> >> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083)
> >> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563)
> >> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123)
> >> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893)
> >> Uninverting(_s(5.5.1):c1969016) Uninverting(_t(5.5.1):c1918535)
> >> Uninverting(_u(5.5.1):c1924194) Uninverting(_w(5.5.1):c1660229)
> >> Uninverting(_x(5.5.1):c49355) Uninverting(_y(5.5.1):c150056)
> >> Uninverting(_z(5.5.1):c63319) Uninverting(_10(5.5.1):c80520)
> >> Uninverting(_11(5.5.1):c92203) Uninverting(_12(5.5.1):c96290)
> >> Uninverting(_13(5.5.1):c24621) Uninverting(_14(5.5.1):c94896)
> >> Uninverting(_15(5.5.1):c8987) Uninverting(_16(5.5.1):c12045)
> >> Uninverting(_17(5.5.1):c10943) Uninverting(_18(5.5.1):c7357)
> >> Uninverting(_19(5.5.1):c7519) Uninverting(_1a(5.5.1):c2426)
> >> Uninverting(_1b(5.5.1):c2700) Uninverting(_1c(5.5.1):c3336)
> >> Uninverting(_1d(5.5.1):c5069) Uninverting(_1e(5.5.1):c3707)
> >> Uninverting(_1f(5.5.1):c2391) Uninverting(_1g(5.5.1):c1389)
> >> Uninverting(_1h(5.5.1):c37) Uninverting(_1i(5.5.1):c42)
> >> Uninverting(_1j(5.5.1):c52)))}
> >> INFO  - 2017-08-23 21:58:28.804; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.update.processor.LogUpdateProcessorFactory$
> LogUpdateProcessor;
> >> [COLLECTION_shard2_1_replica1]  webapp=/solr path=/update
> >> params={update.distrib=FROMLEADER&update.chain=add-
> unknown-fields-to-the-schema&waitSearcher=true&
> openSearcher=true&commit=true&softCommit=true&distrib.from=
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_
> replica1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=}
> >> 0 1728
> >> INFO  - 2017-08-23 21:58:37.739; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.search.
> SolrIndexSearcher;
> >> Opening Searcher@6c639fa[COLLECTION_shard2_0_replica1] main
> >> INFO  - 2017-08-23 21:58:37.790; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
> >> INFO  - 2017-08-23 21:58:37.791; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener sending requests to Searcher@6c639fa[COLLECTION_
> shard2_0_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader(
> Uninverting(_0(5.5.1):C1334281)
> >> Uninverting(_1(5.5.1):C1289239) Uninverting(_l(5.5.1):C3935124)
> >> Uninverting(_r(5.5.1):C4462687) Uninverting(_4(5.5.1):C1562371)
> >> Uninverting(_f(5.5.1):C3882129) Uninverting(_i(5.5.1):C3961582)
> >> Uninverting(_o(5.5.1):C3940869) Uninverting(_c(5.5.1):C3831140)
> >> Uninverting(_g(5.5.1):c1143897) Uninverting(_j(5.5.1):c1145692)
> >> Uninverting(_m(5.5.1):c1156140) Uninverting(_n(5.5.1):c1599534)
> >> Uninverting(_p(5.5.1):c1856273) Uninverting(_q(5.5.1):c1448664)
> >> Uninverting(_s(5.5.1):c1971801) Uninverting(_v(5.5.1):c3849462)
> >> Uninverting(_w(5.5.1):c1658703) Uninverting(_x(5.5.1):c50020)
> >> Uninverting(_y(5.5.1):c150398) Uninverting(_z(5.5.1):c63126)
> >> Uninverting(_10(5.5.1):c80664) Uninverting(_11(5.5.1):c92022)
> >> Uninverting(_12(5.5.1):c96419) Uninverting(_13(5.5.1):c24559)
> >> Uninverting(_14(5.5.1):c94825) Uninverting(_15(5.5.1):c8860)
> >> Uninverting(_16(5.5.1):c11971) Uninverting(_17(5.5.1):c10952)
> >> Uninverting(_18(5.5.1):c7432) Uninverting(_19(5.5.1):c7526)
> >> Uninverting(_1a(5.5.1):c2343) Uninverting(_1b(5.5.1):c2676)
> >> Uninverting(_1c(5.5.1):c3202) Uninverting(_1d(5.5.1):c4974)
> >> Uninverting(_1e(5.5.1):c3736) Uninverting(_1f(5.5.1):c2345)
> >> Uninverting(_1g(5.5.1):c1385) Uninverting(_1h(5.5.1):c43)
> >> Uninverting(_1i(5.5.1):c39) Uninverting(_1j(5.5.1):c42)))}
> >> INFO  - 2017-08-23 21:58:37.836; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener done.
> >> INFO  - 2017-08-23 21:58:37.836; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrCore;
> >> [COLLECTION_shard2_0_replica1] Registered new searcher Searcher@6c639fa
> [COLLECTION_shard2_0_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader(
> Uninverting(_0(5.5.1):C1334281)
> >> Uninverting(_1(5.5.1):C1289239) Uninverting(_l(5.5.1):C3935124)
> >> Uninverting(_r(5.5.1):C4462687) Uninverting(_4(5.5.1):C1562371)
> >> Uninverting(_f(5.5.1):C3882129) Uninverting(_i(5.5.1):C3961582)
> >> Uninverting(_o(5.5.1):C3940869) Uninverting(_c(5.5.1):C3831140)
> >> Uninverting(_g(5.5.1):c1143897) Uninverting(_j(5.5.1):c1145692)
> >> Uninverting(_m(5.5.1):c1156140) Uninverting(_n(5.5.1):c1599534)
> >> Uninverting(_p(5.5.1):c1856273) Uninverting(_q(5.5.1):c1448664)
> >> Uninverting(_s(5.5.1):c1971801) Uninverting(_v(5.5.1):c3849462)
> >> Uninverting(_w(5.5.1):c1658703) Uninverting(_x(5.5.1):c50020)
> >> Uninverting(_y(5.5.1):c150398) Uninverting(_z(5.5.1):c63126)
> >> Uninverting(_10(5.5.1):c80664) Uninverting(_11(5.5.1):c92022)
> >> Uninverting(_12(5.5.1):c96419) Uninverting(_13(5.5.1):c24559)
> >> Uninverting(_14(5.5.1):c94825) Uninverting(_15(5.5.1):c8860)
> >> Uninverting(_16(5.5.1):c11971) Uninverting(_17(5.5.1):c10952)
> >> Uninverting(_18(5.5.1):c7432) Uninverting(_19(5.5.1):c7526)
> >> Uninverting(_1a(5.5.1):c2343) Uninverting(_1b(5.5.1):c2676)
> >> Uninverting(_1c(5.5.1):c3202) Uninverting(_1d(5.5.1):c4974)
> >> Uninverting(_1e(5.5.1):c3736) Uninverting(_1f(5.5.1):c2345)
> >> Uninverting(_1g(5.5.1):c1385) Uninverting(_1h(5.5.1):c43)
> >> Uninverting(_1i(5.5.1):c39) Uninverting(_1j(5.5.1):c42)))}
> >> INFO  - 2017-08-23 21:58:37.836; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.update.processor.LogUpdateProcessorFactory$
> LogUpdateProcessor;
> >> [COLLECTION_shard2_0_replica1]  webapp=/solr path=/update
> >> params={update.distrib=FROMLEADER&update.chain=add-
> unknown-fields-to-the-schema&waitSearcher=true&
> openSearcher=true&commit=true&softCommit=true&distrib.from=
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_
> replica1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=}
> >> 0 10806
> >> INFO  - 2017-08-23 21:58:38.055; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1]
> >> org.apache.solr.update.processor.LogUpdateProcessorFactory$
> LogUpdateProcessor;
> >> [COLLECTION_shard2_replica1]  webapp=/solr path=/update
> >> params={waitSearcher=true&commit=true&softCommit=true&
> wt=javabin&version=2}{commit=}
> >> 0 11828
> >> INFO  - 2017-08-23 21:58:38.105; [   ]
> >> org.apache.solr.common.cloud.SolrZkClient; makePath:
> >> /overseer/collection-map-completed/mn-1000
> >> INFO  - 2017-08-23 21:58:38.152; [   ]
> >> org.apache.solr.cloud.OverseerTaskQueue; Response ZK path:
> >> /overseer/collection-queue-work/qnr-0000000025 doesn't exist.
> Requestor
> >> may have disconnected from ZooKeeper
> >> INFO  - 2017-08-23 21:58:38.152; [   ]
> >> org.apache.solr.cloud.OverseerTaskProcessor$Runner; Overseer Collection
> >> Message Handler: Message id:/overseer/collection-queue-
> work/qn-0000000025
> >> complete,
> >> response:{success={xxx.xx.xx.xxx:8983_solr={responseHeader=
> {status=0,QTime=1}},xxx.xx.xx.xxx:8983_solr={responseHeader=
> {status=0,QTime=1}},xxx.xx.xx.xxx:8983_solr={responseHeader=
> {status=0,QTime=1}},xxx.xx.xx.xxx:8983_solr={responseHeader=
> {status=0,QTime=0}},xxx.xx.xx.xxx:8983_solr={responseHeader=
> {status=0,QTime=0}},xxx.xx.xx.xxx:8983_solr={responseHeader=
> {status=0,QTime=43}},xxx.xx.xx.xxx:8983_solr={responseHeader={status=0,
> QTime=131}},
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_
> replica1/={responseHeader={status=0,QTime=11871}}},1000455258538499159={
> responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> >> 1000455258538499159 webapp=null path=/admin/cores
> >> params={async=1000455258538499159&qt=/admin/
> cores&collection.configName=COLLECTION&name=COLLECTION_
> shard2_0_replica1&action=CREATE&collection=COLLECTION&
> shard=shard2_0&wt=javabin&version=2}
> >> status=0
> >> QTime=1},1000455261568273501={responseHeader={status=0,
> QTime=0},STATUS=completed,Response=TaskId:
> >> 1000455261568273501 webapp=null path=/admin/cores
> >> params={async=1000455261568273501&qt=/admin/
> cores&collection.configName=COLLECTION&name=COLLECTION_
> shard2_1_replica1&action=CREATE&collection=COLLECTION&
> shard=shard2_1&wt=javabin&version=2}
> >> status=0
> >> QTime=1},1000455263597970918={responseHeader={status=0,
> QTime=1},STATUS=completed,Response=TaskId:
> >> 1000455263597970918 webapp=null path=/admin/cores
> >> params={nodeName=xxx.xx.xx.xxx:8983_solr&core=COLLECTION_
> shard2_1_replica1&async=1000455263597970918&qt=/admin/
> cores&coreNodeName=core_node6&action=PREPRECOVERY&checkLive=
> true&state=active&onlyIfLeader=true&wt=javabin&version=2}
> >> status=0
> >> QTime=1},1000455263617138064={responseHeader={status=0,
> QTime=0},STATUS=completed,Response=TaskId:
> >> 1000455263617138064 webapp=null path=/admin/cores
> >> params={core=COLLECTION_shard2_replica1&async=
> 1000455263617138064&qt=/admin/cores&action=SPLIT&targetCore=
> COLLECTION_shard2_0_replica1&targetCore=COLLECTION_shard2_
> 1_replica1&wt=javabin&version=2}
> >> status=0
> >> QTime=0},1000487066831162147={responseHeader={status=0,
> QTime=0},STATUS=completed,Response=TaskId:
> >> 1000487066831162147 webapp=null path=/admin/cores
> >> params={async=1000487066831162147&qt=/admin/
> cores&name=COLLECTION_shard2_1_replica1&action=REQUESTAPPLYUPDATES&wt=
> javabin&version=2}
> >> status=0 QTime=131}}
> >> INFO  - 2017-08-23 21:58:38.156; [   ]
> >> org.apache.solr.common.cloud.SolrZkClient; makePath:
> >> /overseer/collection-map-running/mn-1001
> >> INFO  - 2017-08-23 21:58:38.157; [   ]
> >> org.apache.solr.cloud.OverseerTaskProcessor; Overseer Collection
> Message
> >> Handler: Get the message id:/overseer/collection-queue-
> work/qn-0000000026
> >> message:{
> >>   "collection":"COLLECTION",
> >>   "shard":"shard3",
> >>   "async":"1001",
> >>   "operation":"splitshard"}
> >> INFO  - 2017-08-23 21:58:38.202; [   ]
> >> org.apache.solr.cloud.OverseerCollectionMessageHandler;
> >> OverseerCollectionMessageHandler.processMessage : splitshard , {
> >>   "collection":"COLLECTION",
> >>   "shard":"shard3",
> >>   "async":"1001",
> >>   "operation":"splitshard"}
> >> INFO  - 2017-08-23 21:58:38.246; [   ]
> >> org.apache.solr.cloud.OverseerCollectionMessageHandler; Split shard
> invoked
> >> ............
> >> ............
> >> INFO  - 2017-08-24 07:41:18.387; [   ]
> >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state shard3
> to
> >> inactive
> >> INFO  - 2017-08-24 07:41:18.387; [   ]
> >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state
> shard3_0 to
> >> active
> >> INFO  - 2017-08-24 07:41:18.387; [   ]
> >> org.apache.solr.cloud.overseer.SliceMutator; Update shard state
> shard3_1 to
> >> active
> >> INFO  - 2017-08-24 07:41:18.391; [   ]
> >> org.apache.solr.cloud.overseer.ZkStateWriter; going to
> update_collection
> >> /collections/COLLECTION/state.json
> >>  version: 2700
> >> INFO  - 2017-08-24 07:41:18.395; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster
> state
> >> change: [WatchedEvent state:Syn
> >> cConnected type:NodeDataChanged path:/collections/COLLECTION/
> state.json]
> >> for collection [COLLECTION] has occurred - updating... (live nodes
> size: [8]
> >> )
> >> INFO  - 2017-08-24 07:41:18.395; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader; Updating data for
> [COLLECTION]
> >> from [2700] to [2701]
> >> INFO  - 2017-08-24 07:41:38.169; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1] org.apache.solr.update.
> DirectUpdateHandler2;
> >> start
> >> commit{,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=true,prepareCommit=false}
> >> INFO  - 2017-08-24 07:41:38.169; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.update.DirectUpdateHandler2; start
> >> commit{,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=true,prepareCommit=false}
> >> INFO  - 2017-08-24 07:41:38.253; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.update.DirectUpdateHandler2; start
> >> commit{,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=true,prepareCommit=false}
> >> INFO  - 2017-08-24 07:41:38.793; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrCore;
> >> SolrIndexSearcher has not changed - not re-opening:
> >> org.apache.solr.search.SolrIndexSearcher
> >> INFO  - 2017-08-24 07:41:38.793; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrCore;
> >> SolrIndexSearcher has not changed - not re-opening:
> >> org.apache.solr.search.SolrIndexSearcher
> >> INFO  - 2017-08-24 07:41:38.837; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
> >> INFO  - 2017-08-24 07:41:38.976; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1] org.apache.solr.update.
> DirectUpdateHandler2;
> >> end_commit_flush
> >> INFO  - 2017-08-24 07:41:38.976; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.update.processor.LogUpdateProc
> >> essorFactory$LogUpdateProcessor; [COLLECTION_shard2_0_replica1]
> >>  webapp=/solr path=/update
> >> params={update.distrib=FROMLEADER&update.chain=add-unkn
> >>
> >> own-fields-to-the-schema&waitSearcher=true&
> openSearcher=true&commit=true&softCommit=true&distrib.from=
> >> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_sh
> >> ard3_replica1/&commit_end_point=true&wt=javabin&version=
> 2&expungeDeletes=false}{commit=}
> >> 0 807
> >> INFO  - 2017-08-24 07:41:38.976; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1]
> >> org.apache.solr.update.processor.LogUpdateProcesso
> >> rFactory$LogUpdateProcessor; [COLLECTION_shard2_replica1]  webapp=/solr
> >> path=/update params={update.distrib=FROMLEADER&update.chain=add-
> unknown-fi
> >>
> >> elds-to-the-schema&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=true&distrib.from=
> >> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_shard3_r
> >> eplica1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=}
> >> 0 857
> >> INFO  - 2017-08-24 07:41:53.455; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.search.
> SolrIndexSearcher;
> >> Open
> >> ing Searcher@2b1e0336[COLLECTION_shard2_1_replica1] main
> >> INFO  - 2017-08-24 07:41:53.503; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.update.DirectUpdateHandler2; end_commit_flush
> >> INFO  - 2017-08-24 07:41:53.503; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener sending requests to Searcher@2b1e0336[COLLECTION_
> shard2_1_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader(
> Uninverting(_0(5.5.1):C1332771)
> >> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638)
> >> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033)
> >> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611)
> >> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083)
> >> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563)
> >> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123)
> >> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893)
> >> Uninverting(_s(5.5.1):c1969016) Uninverting(_v(5.5.1):c3842729)
> >> Uninverting(_w(5.5.1):c1660229) Uninverting(_x(5.5.1):c49355)
> >> Uninverting(_y(5.5.1):c150056) Uninverting(_z(5.5.1):c63319)
> >> Uninverting(_10(5.5.1):c80520) Uninverting(_11(5.5.1):c92203)
> >> Uninverting(_12(5.5.1):c96290) Uninverting(_13(5.5.1):c24621)
> >> Uninverting(_14(5.5.1):c94896) Uninverting(_15(5.5.1):c8987)
> >> Uninverting(_16(5.5.1):c12045) Uninverting(_17(5.5.1):c10943)
> >> Uninverting(_18(5.5.1):c7357) Uninverting(_19(5.5.1):c7519)
> >> Uninverting(_1a(5.5.1):c2426) Uninverting(_1b(5.5.1):c2700)
> >> Uninverting(_1c(5.5.1):c3336) Uninverting(_1d(5.5.1):c5069)
> >> Uninverting(_1e(5.5.1):c3707) Uninverting(_1f(5.5.1):c2391)
> >> Uninverting(_1g(5.5.1):c1389) Uninverting(_1h(5.5.1):c37)
> >> Uninverting(_1i(5.5.1):c42) Uninverting(_1j(5.5.1):c52)))}
> >> INFO  - 2017-08-24 07:41:53.547; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener done.
> >> INFO  - 2017-08-24 07:41:53.547; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.SolrCore;
> >> [COLLECTION_shard2_1_replica1] Registered new searcher Searcher@2b1e0336
> [COLLECTION_shard2_1_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader(
> Uninverting(_0(5.5.1):C1332771)
> >> Uninverting(_1(5.5.1):C1290745) Uninverting(_m(5.5.1):C3937638)
> >> Uninverting(_r(5.5.1):C4458900) Uninverting(_4(5.5.1):C1563033)
> >> Uninverting(_g(5.5.1):C3875170) Uninverting(_j(5.5.1):C3956611)
> >> Uninverting(_o(5.5.1):C3938682) Uninverting(_c(5.5.1):C3836083)
> >> Uninverting(_f(5.5.1):c1146182) Uninverting(_i(5.5.1):c1147563)
> >> Uninverting(_l(5.5.1):c1157380) Uninverting(_n(5.5.1):c1598123)
> >> Uninverting(_p(5.5.1):c1854952) Uninverting(_q(5.5.1):c1447893)
> >> Uninverting(_s(5.5.1):c1969016) Uninverting(_v(5.5.1):c3842729)
> >> Uninverting(_w(5.5.1):c1660229) Uninverting(_x(5.5.1):c49355)
> >> Uninverting(_y(5.5.1):c150056) Uninverting(_z(5.5.1):c63319)
> >> Uninverting(_10(5.5.1):c80520) Uninverting(_11(5.5.1):c92203)
> >> Uninverting(_12(5.5.1):c96290) Uninverting(_13(5.5.1):c24621)
> >> Uninverting(_14(5.5.1):c94896) Uninverting(_15(5.5.1):c8987)
> >> Uninverting(_16(5.5.1):c12045) Uninverting(_17(5.5.1):c10943)
> >> Uninverting(_18(5.5.1):c7357) Uninverting(_19(5.5.1):c7519)
> >> Uninverting(_1a(5.5.1):c2426) Uninverting(_1b(5.5.1):c2700)
> >> Uninverting(_1c(5.5.1):c3336) Uninverting(_1d(5.5.1):c5069)
> >> Uninverting(_1e(5.5.1):c3707) Uninverting(_1f(5.5.1):c2391)
> >> Uninverting(_1g(5.5.1):c1389) Uninverting(_1h(5.5.1):c37)
> >> Uninverting(_1i(5.5.1):c42) Uninverting(_1j(5.5.1):c52)))}
> >> INFO  - 2017-08-24 07:41:56.237; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.update.processor.LogUpdateProcessorFactory$
> LogUpdateProcessor;
> >> [COLLECTION_shard2_1_replica1]  webapp=/solr path=/update
> >> params={update.distrib=FROMLEADER&update.chain=add-
> unknown-fields-to-the-schema&waitSearcher=true&
> openSearcher=true&commit=true&softCommit=true&distrib.from=
> >> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_shard3_
> replica1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=}
> >> 0 18117
> >> INFO  - 2017-08-24 07:42:24.803; [   ]
> >> org.apache.solr.common.cloud.SolrZkClient; makePath:
> >> /overseer/collection-map-completed/mn-1001
> >> INFO  - 2017-08-24 07:42:24.856; [   ]
> >> org.apache.solr.cloud.OverseerTaskProcessor$Runner; Overseer Collection
> >> Message Handler: Message id:/overseer/collection-queue-
> work/qn-0000000026
> >> complete,
> >> response:{success={xxx.xx.xx.xxx:8984_solr={responseHeader=
> {status=0,QTime=4}},xxx.xx.xx.xxx:8984_solr={responseHeader=
> {status=0,QTime=0}},xxx.xx.xx.xxx:8984_solr={responseHeader=
> {status=0,QTime=1}},xxx.xx.xx.xxx:8984_solr={responseHeader=
> {status=0,QTime=1}},xxx.xx.xx.xxx:8984_solr={responseHeader=
> {status=0,QTime=1}},xxx.xx.xx.xxx:8984_solr={responseHeader=
> {status=0,QTime=0}},xxx.xx.xx.xxx:8984_solr={responseHeader=
> {status=0,QTime=189}},
> >> http://xxx.xx.xx.xxx:8984/solr/COLLECTION_shard3_
> replica1/={responseHeader={status=0,QTime=66236}}},1001487080480985685={
> responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId:
> >> 1001487080480985685 webapp=null path=/admin/cores
> >> params={async=1001487080480985685&qt=/admin/
> cores&collection.configName=COLLECTION&name=COLLECTION_
> shard3_0_replica1&action=CREATE&collection=COLLECTION&
> shard=shard3_0&wt=javabin&version=2}
> >> status=0
> >> QTime=4},1001487084675100247={responseHeader={status=0,
> QTime=0},STATUS=completed,Response=TaskId:
> >> 1001487084675100247 webapp=null path=/admin/cores
> >> params={async=1001487084675100247&qt=/admin/
> cores&collection.configName=COLLECTION&name=COLLECTION_
> shard3_1_replica1&action=CREATE&collection=COLLECTION&
> shard=shard3_1&wt=javabin&version=2}
> >> status=0
> >> QTime=0},1001487086698326060={responseHeader={status=0,
> QTime=0},STATUS=completed,Response=TaskId:
> >> 1001487086698326060 webapp=null path=/admin/cores
> >> params={nodeName=xxx.xx.xx.xxx:8984_solr&core=COLLECTION_
> shard3_1_replica1&async=1001487086698326060&qt=/admin/
> cores&coreNodeName=core_node8&action=PREPRECOVERY&checkLive=
> true&state=active&onlyIfLeader=true&wt=javabin&version=2}
> >> status=0
> >> QTime=1},1001487086756758564={responseHeader={status=0,
> QTime=0},STATUS=completed,Response=TaskId:
> >> 1001487086756758564 webapp=null path=/admin/cores
> >> params={core=COLLECTION_shard3_replica1&async=
> 1001487086756758564&qt=/admin/cores&action=SPLIT&targetCore=
> COLLECTION_shard3_0_replica1&targetCore=COLLECTION_shard3_
> 1_replica1&wt=javabin&version=2}
> >> status=0
> >> QTime=1},1001522038777271446={responseHeader={status=0,
> QTime=0},STATUS=completed,Response=TaskId:
> >> 1001522038777271446 webapp=null path=/admin/cores
> >> params={async=1001522038777271446&qt=/admin/
> cores&name=COLLECTION_shard3_1_replica1&action=REQUESTAPPLYUPDATES&wt=
> javabin&version=2}
> >> status=0 QTime=189}}
> >
> >
> > *Log messages during next restart:*
> >
> >> INFO  - 2017-08-24 08:52:37.821; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.handler.
> ReplicationHandler;
> >> Co
> >> mmits will be reserved for  10000
> >> INFO  - 2017-08-24 08:52:37.826; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.
> SolrDeletionPolicy;
> >> SolrD
> >> eletionPolicy.onInit: commits: num=1
> >>         commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solrcloud/
> solr-5.5.1/example/cloud/node1/solr/COLLECTION_shard2_
> 0_replica1/data/index
> >> lo
> >> ckFactory=org.apache.lucene.store.NativeFSLockFactory@725fe252;
> >> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
> >> INFO  - 2017-08-24 08:52:37.826; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.
> SolrDeletionPolicy;
> >> newes
> >> t commit generation = 1
> >> INFO  - 2017-08-24 08:52:37.827; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.search.
> SolrIndexSearcher;
> >> Open
> >> ing Searcher@5c49571c[COLLECTION_shard2_0_replica1] main
> >> INFO  - 2017-08-24 08:52:37.827; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 1, message = {
> >>   "core":"COLLECTION-digital-archive_shard1_replica1",
> >>   "core_node_name":"core_node1",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8983/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8983_solr",
> >>   "numShards":"1",
> >>   "state":"active",
> >>   "shard":"shard1",
> >>   "collection":"COLLECTION-digital-archive",
> >>   "operation":"state"} current state version: 0
> >> INFO  - 2017-08-24 08:52:37.827; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud.
> ZkStateReader;
> >> Load collection config from: [/collections/COLLECTION]
> >> INFO  - 2017-08-24 08:52:37.827; [   ]
> >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state numShards=1
> >> message={
> >>   "core":"COLLECTION-digital-archive_shard1_replica1",
> >>   "core_node_name":"core_node1",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8983/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8983_solr",
> >>   "numShards":"1",
> >>   "state":"active",
> >>   "shard":"shard1",
> >>   "collection":"COLLECTION-digital-archive",
> >>   "operation":"state"}
> >> INFO  - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud.
> ZkStateReader;
> >> path=[/collections/COLLECTION] [configName]=[COLLECTION] specified
> config
> >> exists in ZooKeeper
> >> INFO  - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage; Setting up ZooKeeper-based
> >> storage for the RestManager with znodeBase: /configs/COLLECTION
> >> INFO  - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrDeletionPolicy;
> >> SolrDeletionPolicy.onInit: commits: num=1
> >>         commit{dir=NRTCachingDirectory(MMapDirectory@/opt/solrdata/
> index
> >> lockFactory=org.apache.lucene.store.NativeFSLockFactory@725fe252;
> >> maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_33sj,generation=
> 144883}
> >> INFO  - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.UpdateLog;
> Looking
> >> up max value of version field to seed version buckets
> >> INFO  - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2 r:core_node1
> >> x:COLLECTION_shard2_replica1] org.apache.solr.core.SolrDeletionPolicy;
> >> newest commit generation = 144883
> >> INFO  - 2017-08-24 08:52:37.828; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener sending requests to Searcher@269e2d59[COLLECTION_
> shard2_1_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> >> INFO  - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener done.
> >> INFO  - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.handler.component.SpellCheckComponent$
> SpellCheckerListener;
> >> Loading spell index for spellchecker: default
> >> INFO  - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.VersionInfo;
> >> Refreshing highest value of _version_ for 65536 version buckets from
> index
> >> INFO  - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.SolrCore;
> >> [COLLECTION_shard2_1_replica1] Registered new searcher Searcher@269e2d59
> [COLLECTION_shard2_1_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> >> INFO  - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.VersionInfo; No
> >> terms found for _version_, cannot seed version bucket highest value from
> >> index
> >> INFO  - 2017-08-24 08:52:37.829; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO;
> Configured
> >> ZooKeeperStorageIO with znodeBase: /configs/COLLECTION
> >> INFO  - 2017-08-24 08:52:37.830; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.RestManager;
> >> Initializing RestManager with initArgs: {}
> >> INFO  - 2017-08-24 08:52:37.830; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.UpdateLog;
> Could not
> >> find max version in index or recent updates, using new clock
> >> 1576601919506350080
> >> INFO  - 2017-08-24 08:52:37.830; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage; Reading _rest_managed.json
> >> using ZooKeeperStorageIO:path=/configs/COLLECTION
> >> INFO  - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Read 33
> >> bytes from znode /configs/COLLECTION/_rest_managed.json
> >> INFO  - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage; Loaded LinkedHashMap at
> path
> >> _rest_managed.json using ZooKeeperStorageIO:path=/configs/COLLECTION
> >> INFO  - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> >> Loaded initArgs {} for /rest/managed
> >> INFO  - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.RestManager;
> >> Initializing 2 registered ManagedResources
> >> INFO  - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage; Reading
> >> _schema_analysis_stopwords_english.json using
> >> ZooKeeperStorageIO:path=/configs/COLLECTION
> >> INFO  - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.update.UpdateLog; Took
> >> 3.0ms to seed version buckets with highest version 1576601919506350080
> >> INFO  - 2017-08-24 08:52:37.832; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController;
> watch
> >> zkdir /configs/COLLECTION
> >> INFO  - 2017-08-24 08:52:37.833; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.core.CoreContainer;
> >> registering core: COLLECTION_shard2_1_replica1
> >> INFO  - 2017-08-24 08:52:37.833; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Read
> 450
> >> bytes from znode /configs/COLLECTION/_schema_
> analysis_stopwords_english.json
> >> INFO  - 2017-08-24 08:52:37.834; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController;
> >> Register replica - core:COLLECTION_shard2_1_replica1 address:
> >> http://xxx.xx.xx.xxx:8983/solr collection:COLLECTION shard:shard2_1
> >> INFO  - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage; Loaded LinkedHashMap at
> path
> >> _schema_analysis_stopwords_english.json using
> >> ZooKeeperStorageIO:path=/configs/COLLECTION
> >> INFO  - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> >> Loaded initArgs {ignoreCase=true} for /schema/analysis/stopwords/
> english
> >> INFO  - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.schema.analysis.ManagedWordSetResource; Loaded 35
> >> words for /schema/analysis/stopwords/english
> >> INFO  - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> >> Notified 1 observers of /schema/analysis/stopwords/english
> >> INFO  - 2017-08-24 08:52:37.835; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage; Reading
> >> _schema_analysis_synonyms_english.json using
> >> ZooKeeperStorageIO:path=/configs/COLLECTION
> >> INFO  - 2017-08-24 08:52:37.837; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.LeaderElector;
> Joined
> >> leadership election with path:
> >> /collections/COLLECTION/leader_elect/shard2_1/
> election/242652799835373569-core_node6-n_0000000003
> >> INFO  - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage$ZooKeeperStorageIO; Read
> 172
> >> bytes from znode /configs/COLLECTION/_schema_
> analysis_synonyms_english.json
> >> INFO  - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.ManagedResourceStorage; Loaded LinkedHashMap at
> path
> >> _schema_analysis_synonyms_english.json using
> >> ZooKeeperStorageIO:path=/configs/COLLECTION
> >> INFO  - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader
> >> process for shard=shard2_1 and weAreReplacement=false and
> >> leaderVoteWait=180000
> >> INFO  - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> >> Loaded initArgs {ignoreCase=true,format=solr} for
> >> /schema/analysis/synonyms/english
> >> INFO  - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.rest.schema.analysis.ManagedSynonymFilterFactory$
> SynonymManager;
> >> Loaded 3 synonym mappings for /schema/analysis/synonyms/english
> >> INFO  - 2017-08-24 08:52:37.838; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.rest.ManagedResource;
> >> Notified 1 observers of /schema/analysis/synonyms/english
> >> INFO  - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.handler.component.SpellCheckComponent; Initializing
> spell
> >> checkers
> >> INFO  - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.spelling.DirectSolrSpellChecker; init:
> >> {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,
> distanceMeasure=internal,accuracy=0.5,maxEdits=2,
> minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
> >> INFO  - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.handler.component.SpellCheckComponent; No
> queryConverter
> >> defined, using default converter
> >> INFO  - 2017-08-24 08:52:37.839; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud.
> ZkStateReader;
> >> Load collection config from: [/collections/COLLECTION]
> >> INFO  - 2017-08-24 08:52:37.840; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.common.cloud.
> ZkStateReader;
> >> path=[/collections/COLLECTION] [configName]=[COLLECTION] specified
> config
> >> exists in ZooKeeper
> >> INFO  - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas
> found to
> >> continue.
> >> INFO  - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new
> leader -
> >> try and sync
> >> INFO  - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.SyncStrategy;
> Sync
> >> replicas to http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_
> replica1/
> >> INFO  - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.SyncStrategy;
> Sync
> >> Success - now sync replicas to me
> >> INFO  - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.handler.
> ReplicationHandler;
> >> Commits will be reserved for  10000
> >> INFO  - 2017-08-24 08:52:37.843; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.SyncStrategy;
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ has no
> >> replicas
> >> INFO  - 2017-08-24 08:52:37.844; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 1, message = {
> >>   "operation":"leader",
> >>   "shard":"shard2_1",
> >>   "collection":"COLLECTION"} current state version: 0
> >> INFO  - 2017-08-24 08:52:37.844; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContextBase$1; Creating leader
> >> registration node /collections/COLLECTION/leaders/shard2_1/leader after
> >> winning as
> >> /collections/COLLECTION/leader_elect/shard2_1/
> election/242652799835373569-core_node6-n_0000000003
> >> INFO  - 2017-08-24 08:52:37.847; [   ]
> >> org.apache.solr.cloud.overseer.ZkStateWriter; going to
> update_collection
> >> /collections/COLLECTION-digital-archive/state.json version: 902
> >> INFO  - 2017-08-24 08:52:37.850; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster
> state
> >> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> >> path:/collections/COLLECTION-digital-archive/state.json] for collection
> >> [COLLECTION-digital-archive] has occurred - updating... (live nodes
> size:
> >> [1])
> >> INFO  - 2017-08-24 08:52:37.852; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader; Updating data for
> >> [COLLECTION-digital-archive] from [902] to [903]
> >> INFO  - 2017-08-24 08:52:37.852; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader:
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ shard2_1
> >> INFO  - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener sending requests to Searcher@5c49571c[COLLECTION_
> shard2_0_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> >> INFO  - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; We
> are
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ and leader
> >> is http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/
> >> INFO  - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog;
> Looking
> >> up max value of version field to seed version buckets
> >> INFO  - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.
> QuerySenderListener;
> >> QuerySenderListener done.
> >> INFO  - 2017-08-24 08:52:37.854; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; We
> are
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/ and leader
> >> is http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_1_replica1/
> >> INFO  - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog;
> Looking
> >> up max value of version field to seed version buckets
> >> INFO  - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.handler.component.SpellCheckComponent$
> SpellCheckerListener;
> >> Loading spell index for spellchecker: default
> >> INFO  - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.VersionInfo;
> >> Refreshing highest value of _version_ for 65536 version buckets from
> index
> >> INFO  - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.VersionInfo; No
> >> terms found for _version_, cannot seed version bucket highest value from
> >> index
> >> INFO  - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; No
> >> LogReplay needed for core=COLLECTION_shard2_1_replica1 baseURL=
> >> http://xxx.xx.xx.xxx:8983/solr
> >> INFO  - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController; I
> am
> >> the leader, no recovery necessary
> >> INFO  - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog;
> Could not
> >> find max version in index or recent updates, using new clock
> >> 1576601919533613056
> >> INFO  - 2017-08-24 08:52:37.855; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.SolrCore;
> >> [COLLECTION_shard2_0_replica1] Registered new searcher Searcher@5c49571c
> [COLLECTION_shard2_0_replica1]
> >> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
> >> INFO  - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController;
> >> publishing state=active
> >> INFO  - 2017-08-24 08:52:37.856; [c:COLLECTION s:shard2_1 r:core_node6
> >> x:COLLECTION_shard2_1_replica1] org.apache.solr.cloud.ZkController;
> >> numShards not found on descriptor - reading it from system property
> >> INFO  - 2017-08-24 08:52:37.857; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.update.UpdateLog; Took
> >> 2.0ms to seed version buckets with highest version 1576601919533613056
> >> INFO  - 2017-08-24 08:52:37.857; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.core.CoreContainer;
> >> registering core: COLLECTION_shard2_0_replica1
> >> INFO  - 2017-08-24 08:52:37.858; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController;
> >> Register replica - core:COLLECTION_shard2_0_replica1 address:
> >> http://xxx.xx.xx.xxx:8983/solr collection:COLLECTION shard:shard2_0
> >> INFO  - 2017-08-24 08:52:37.862; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.LeaderElector;
> Joined
> >> leadership election with path:
> >> /collections/COLLECTION/leader_elect/shard2_0/
> election/242652799835373569-core_node5-n_0000000003
> >> INFO  - 2017-08-24 08:52:37.864; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContext; Running the leader
> >> process for shard=shard2_0 and weAreReplacement=false and
> >> leaderVoteWait=180000
> >> INFO  - 2017-08-24 08:52:37.867; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 2, message = {
> >>   "operation":"leader",
> >>   "shard":"shard2_1",
> >>   "collection":"COLLECTION",
> >>   "base_url":"http://xxx.xx.xx.xxx:8983/solr",
> >>   "core":"COLLECTION_shard2_1_replica1",
> >>   "state":"active"} current state version: 0
> >> INFO  - 2017-08-24 08:52:37.871; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContext; Enough replicas
> found to
> >> continue.
> >> INFO  - 2017-08-24 08:52:37.871; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContext; I may be the new
> leader -
> >> try and sync
> >> INFO  - 2017-08-24 08:52:37.872; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.SyncStrategy;
> Sync
> >> replicas to http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_
> replica1/
> >> INFO  - 2017-08-24 08:52:37.872; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.SyncStrategy;
> Sync
> >> Success - now sync replicas to me
> >> INFO  - 2017-08-24 08:52:37.872; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.SyncStrategy;
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ has no
> >> replicas
> >> INFO  - 2017-08-24 08:52:37.874; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContextBase$1; Creating leader
> >> registration node /collections/COLLECTION/leaders/shard2_0/leader after
> >> winning as
> >> /collections/COLLECTION/leader_elect/shard2_0/
> election/242652799835373569-core_node5-n_0000000003
> >> INFO  - 2017-08-24 08:52:37.874; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 2, message = {
> >>   "core":"COLLECTION_shard2_1_replica1",
> >>   "core_node_name":"core_node6",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8983/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8983_solr",
> >>   "state":"active",
> >>   "shard":"shard2_1",
> >>   "collection":"COLLECTION",
> >>   "operation":"state"} current state version: 0
> >> INFO  - 2017-08-24 08:52:37.876; [   ]
> >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state
> numShards=null
> >> message={
> >>   "core":"COLLECTION_shard2_1_replica1",
> >>   "core_node_name":"core_node6",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8983/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8983_solr",
> >>   "state":"active",
> >>   "shard":"shard2_1",
> >>   "collection":"COLLECTION",
> >>   "operation":"state"}
> >> INFO  - 2017-08-24 08:52:37.881; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1]
> >> org.apache.solr.cloud.ShardLeaderElectionContext; I am the new leader:
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ shard2_0
> >> INFO  - 2017-08-24 08:52:37.882; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 2, message = {
> >>   "operation":"leader",
> >>   "shard":"shard2_0",
> >>   "collection":"COLLECTION"} current state version: 0
> >> INFO  - 2017-08-24 08:52:37.882; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; We
> are
> >> http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/ and leader
> >> is http://xxx.xx.xx.xxx:8983/solr/COLLECTION_shard2_0_replica1/
> >> INFO  - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; No
> >> LogReplay needed for core=COLLECTION_shard2_0_replica1 baseURL=
> >> http://xxx.xx.xx.xxx:8983/solr
> >> INFO  - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController; I
> am
> >> the leader, no recovery necessary
> >> INFO  - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController;
> >> publishing state=active
> >> INFO  - 2017-08-24 08:52:37.883; [c:COLLECTION s:shard2_0 r:core_node5
> >> x:COLLECTION_shard2_0_replica1] org.apache.solr.cloud.ZkController;
> >> numShards not found on descriptor - reading it from system property
> >> INFO  - 2017-08-24 08:52:37.888; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 2, message = {
> >>   "operation":"leader",
> >>   "shard":"shard2_0",
> >>   "collection":"COLLECTION",
> >>   "base_url":"http://xxx.xx.xx.xxx:8983/solr",
> >>   "core":"COLLECTION_shard2_0_replica1",
> >>   "state":"active"} current state version: 0
> >> INFO  - 2017-08-24 08:52:37.894; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 1, message = {
> >>   "core":"COLLECTION_shard2_0_replica1",
> >>   "core_node_name":"core_node5",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8983/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8983_solr",
> >>   "state":"active",
> >>   "shard":"shard2_0",
> >>   "collection":"COLLECTION",
> >>   "operation":"state"} current state version: 0
> >> INFO  - 2017-08-24 08:52:37.894; [   ]
> >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state
> numShards=null
> >> message={
> >>   "core":"COLLECTION_shard2_0_replica1",
> >>   "core_node_name":"core_node5",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8983/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8983_solr",
> >>   "state":"active",
> >>   "shard":"shard2_0",
> >>   "collection":"COLLECTION",
> >>   "operation":"state"}
> >> INFO  - 2017-08-24 08:52:38.000; [   ]
> >> org.apache.solr.cloud.overseer.ZkStateWriter; going to
> update_collection
> >> /collections/COLLECTION/state.json version: 2721
> >> INFO  - 2017-08-24 08:52:38.005; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster
> state
> >> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> >> path:/collections/COLLECTION/state.json] for collection [COLLECTION]
> has
> >> occurred - updating... (live nodes size: [1])
> >> INFO  - 2017-08-24 08:52:38.006; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader; Updating data for
> [COLLECTION]
> >> from [2721] to [2722]
> >> INFO  - 2017-08-24 08:52:40.805; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher; A live node
> >> change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged
> >> path:/live_nodes], has occurred - updating... (live nodes size: [1])
> >> INFO  - 2017-08-24 08:52:40.931; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 1, message = {
> >>   "core":"COLLECTION1_shard4_replica1",
> >>   "core_node_name":"core_node3",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8984/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8984_solr",
> >>   "numShards":"4",
> >>   "state":"down",
> >>   "shard":"shard4",
> >>   "collection":"COLLECTION1",
> >>   "operation":"state"} current state version: 0
> >> INFO  - 2017-08-24 08:52:40.931; [   ]
> >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state numShards=4
> >> message={
> >>   "core":"COLLECTION1_shard4_replica1",
> >>   "core_node_name":"core_node3",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8984/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8984_solr",
> >>   "numShards":"4",
> >>   "state":"down",
> >>   "shard":"shard4",
> >>   "collection":"COLLECTION1",
> >>   "operation":"state"}
> >> INFO  - 2017-08-24 08:52:40.934; [   ]
> >> org.apache.solr.cloud.overseer.ZkStateWriter; going to
> update_collection
> >> /collections/COLLECTION1/state.json version: 429
> >> INFO  - 2017-08-24 08:52:40.935; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster
> state
> >> change: [WatchedEvent state:SyncConnected type:NodeDataChanged
> >> path:/collections/COLLECTION1/state.json] for collection [COLLECTION1]
> has
> >> occurred - updating... (live nodes size: [2])
> >> INFO  - 2017-08-24 08:52:40.936; [   ]
> >> org.apache.solr.common.cloud.ZkStateReader; Updating data for
> [COLLECTION1]
> >> from [429] to [430]
> >> INFO  - 2017-08-24 08:52:40.940; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 4, message = {
> >>   "core":"COLLECTION_shard3_0_replica1",
> >>   "core_node_name":"core_node7",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8984/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8984_solr",
> >>   "state":"down",
> >>   "shard":"shard3_0",
> >>   "collection":"COLLECTION",
> >>   "operation":"state"} current state version: 0
> >> INFO  - 2017-08-24 08:52:40.940; [   ]
> >> org.apache.solr.cloud.overseer.ReplicaMutator; Update state
> numShards=null
> >> message={
> >>   "core":"COLLECTION_shard3_0_replica1",
> >>   "core_node_name":"core_node7",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8984/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8984_solr",
> >>   "state":"down",
> >>   "shard":"shard3_0",
> >>   "collection":"COLLECTION",
> >>   "operation":"state"}
> >> INFO  - 2017-08-24 08:52:40.946; [   ]
> >> org.apache.solr.cloud.Overseer$ClusterStateUpdater; processMessage:
> >> queueSize: 0, message = {
> >>   "core":"COLLECTION_sandbox_shard4_replica1",
> >>   "core_node_name":"core_node4",
> >>   "roles":null,
> >>   "base_url":"http://xxx.xx.xx.xxx:8984/solr",
> >>   "node_name":"xxx.xx.xx.xxx:8984_solr",
> >>   "numShards":"4",
> >>   "state":"down",
> >>   "shard":"shard4",
> >
> >
> >
> > Thanks,
> > Vanniarajan
> >
> > On Thu, Aug 24, 2017 at 8:36 PM, Binoy Dalal <binoydalal93@gmail.com>
> wrote:
> >
> >> Vanniarajan,
> >> Do you see errors in your solr logs when you reboot? If so paste them
> here.
> >> Are both the shards up?
> >> Have you checked the data directory for both and confirmed that the
> data is
> >> gone?
> >>
> >> On Thu 24 Aug, 2017, 18:04 Vannia Rajan <vannia@knackforge.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm facing weird issues on using SPLITSHARD on fairly large shards
> (150GB
> >> > shard size), using SOLR 5.5.1 in cloud mode (4 nodes, 1 shard per
> node).
> >> >
> >> > After issuing SPLITSHARD, I successfully get the sub-shards activated
> and
> >> > my parent shard set "inactive". The newly split shards counts equal
> the
> >> > parent's counts and all works well. The issue is when I restart SOLR,
> the
> >> > newly split shards gets reduced to 71KB and 0 documents.
> >> >
> >> > Has anyone faced such issues, and do you think this can be a possible
> bug
> >> > that could be fixed if I upgrade SOLR to 6.6?
> >> >
> >> > Also, I would like to know if there is a way to "rollback" the
> >> SPLITSHARD.
> >> > I would like to reactivate my parent shard with all documents intact
> and
> >> > delete the new sub-shards. I have no idea how I could reactivate a
> shard
> >> > that is now marked as "inactive".
> >> >
> >> > Thanks,
> >> > Vanniarajan
> >> >
> >> --
> >> Regards,
> >> Binoy Dalal
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message