lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noble Paul <noble.p...@gmail.com>
Subject Re: what to try next for replica that will not stay up.
Date Sat, 03 Sep 2016 16:36:49 GMT
pls share your security.json (you can obfuscate sensitive info if any)

On Fri, Sep 2, 2016 at 9:56 PM, Jon Hawkesworth
<jon.hawkesworth@medquist.onmicrosoft.com> wrote:
> A little more info:
>
> Once we start indexing we start seeing a lot of these:
>
> 2016-09-02 12:48:29,794 level=INFO    thread=qtp762152757-17 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50
> 2016-09-02 12:48:29,794 level=INFO    thread=qtp762152757-17 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2
> 2016-09-02 12:48:29,794 level=INFO    thread=qtp762152757-17 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
> java.io.IOException: Committed before 100 Continues
>         at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
>         at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
>         at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Unknown Source)
>
> We clearly have a *lot* of leader_initiated_recovery happening on shard1 - looking at the
>
> cversion attribute the /collections/transcribedReports/leader_initiated_recovery/shardX nodes I can see the versions are as follows
>
> shard1 312
> shard2 18
> shard3 24
> shard4 24
>
> Jon
>
>
> -----Original Message-----
> From: Jon Hawkesworth [mailto:jon.hawkesworth@MEDQUIST.onmicrosoft.com]
> Sent: Friday, September 2, 2016 4:14 PM
> To: solr-user@lucene.apache.org
> Subject: RE: what to try next for replica that will not stay up.
>
> Many thanks for continuing to investigate this, Shalin.
>
> So we were able to bring the replica up for approx 30 minutes after removing the leader-initiated-recovery/shard1 node from zookeeper and restarting all 4 nodes.  However soon after we enabled indexing the replica quickly went into Down state again.
>
> I am not sure about garbage collection on the node.  It hasn't looked busy cpu wise and java threads usually seem to be around 50 when I check resource monitor, but I have included the logging things you asked for below 1/ a few seconds of logging from either side of the 'Put replica core=' (the collection is actually called transcribedReports) shortly after we re-enabled indexing and the replica went into Down state, 2/ clusterstate.json, 3/ the collection's solrconfig.xml.  I think the only changes to default solrconfig.xml are for the updateRequestProcessorChain (ignore commits from clients and auto delete expired documents - is every 30seconds too frequent for this?), and auto commit configuration (hard autocommit 15 seconds, opensearch=false soft autocommit every 15 *minutes*).
>
> I note some 401 Unauthorised request, which is odd since security.xml is applied to the cluster.  Also some 'java.io.IOException: Committed before 100 Continues' exceptions which may or may not be of interest.  I am speculating that the 401s between nodes is stopping updates based on the contents of the transaction log, causing it to fall back to old full replication.  While there are no incoming documents to process it is able to keep up with old full replication, but its too much for it to handle when there are new documents arriving all the time (I estimate the cluster recieves 1-2 documents per second on average during the day).
>
> I can send full log from startup, adding the replica and then turning on ingestion if that would be helpful (3+Mb though, not suitable for a mailing list I think).  Please let me know if you need any further information.
>
> Jon
>
>
>
> 1/ (logging)
>
> 2016-09-02 12:49:06,138 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol003:8983/solr/transcribedReports_shard3_replica3/&wt=javabin&version=2}{add=[711930167 (1544364276970422272)]} 0 22
> 2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
> 2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
> 2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
> 2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
> 2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=6
> 2016-09-02 12:49:10,372 level=INFO    thread=qtp762152757-21 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
> 2016-09-02 12:49:10,388 level=INFO    thread=qtp762152757-21 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
> 2016-09-02 12:49:10,388 level=INFO    thread=qtp762152757-21 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
> 2016-09-02 12:49:10,388 level=INFO    thread=qtp762152757-21 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
> 2016-09-02 12:49:10,388 level=INFO    thread=qtp762152757-21 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=6
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.DocExpirationUpdateProcessorFactory Begining periodic deletion of expired docs
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: anyChanges? numDocsInRam=5 deletes=true hasTickets:false pendingChangesInFullFlush: false
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: nrtIsCurrent: infoVersion matches: false; DW changes: true; BD changes: false
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: flush at getReader
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: startFullFlush
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: anyChanges? numDocsInRam=5 deletes=true hasTickets:false pendingChangesInFullFlush: false
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWFC][autoExpireDocs-18-thread-1]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 4 deleted terms (unique count=4) bytesUsed=692, segment=_jtqb, aborted=false, numDocsInRAM=5, deleteQueue=DWDQ: [ generation: 91 ]]
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flush postings as segment _jtqb numDocs=5
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 1 msec to write norms
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 1 msec to write docValues
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 0 msec to write points
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 1 msec to finish stored fields
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 6 msec to write postings and finish vectors
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 0 msec to write fieldInfos
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: new segment has 0 deleted docs
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: new segment has no vectors; norms; docValues; prox; freqs
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flushedFiles=[_jtqb.fdt, _jtqb.nvm, _jtqb.fnm, _jtqb_Lucene50_0.pos, _jtqb.fdx, _jtqb_Lucene54_0.dvm, _jtqb_Lucene50_0.doc, _jtqb_Lucene50_0.tim, _jtqb_Lucene54_0.dvd, _jtqb_Lucene50_0.tip, _jtqb.nvd]
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flushed codec=Lucene60
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flushed: segment=_jtqb ramUsed=0.269 MB newFlushedSize=0.021 MB docs/MB=243.379
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flush time 11.28663 msec
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: publishFlushedSegment seg-private updates= 4 deleted terms (unique count=0) 1 deleted queries bytesUsed=80
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: flush: push buffered seg private updates:  4 deleted terms (unique count=0) 1 deleted queries bytesUsed=80
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: publishFlushedSegment
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: push deletes  5 deleted terms (unique count=5) 1 deleted queries bytesUsed=1104 segmentPrivate?=false delGen=181 packetCount=1 totBytesUsed=1104
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: push deletes  4 deleted terms (unique count=0) 1 deleted queries bytesUsed=80 segmentPrivate?=true delGen=182 packetCount=2 totBytesUsed=1184
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: publish sets newSegment delGen=182 seg=_jtqb(6.0.1):C5
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][autoExpireDocs-18-thread-1]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5" [25 segments ; isCommit = false]
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][autoExpireDocs-18-thread-1]: delete []
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][autoExpireDocs-18-thread-1]: 0 msec to checkpoint
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: apply all deletes during flush
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: now apply all deletes for all segments maxDoc=1450423
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyDeletes: open segment readers took 0 msec
> 2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyDeletes: infos=[_5tw9(6.0.1):C241588/706:delGen=703, _fb82(6.0.1):C387586/788:delGen=727, _iin7(6.0.1):C491496/573:delGen=526, _i0at(6.0.1):c17859/119:delGen=112, _iy15(6.0.1):c32975/162:delGen=157, _iqm7(6.0.1):c67369/342:delGen=325, _jf5b(6.0.1):c22457/84:delGen=83, _j7ia(6.0.1):c65875/307:delGen=272, _jksn(6.0.1):c60239/75:delGen=70, _jthf(6.0.1):c27514/3:delGen=3, _jmdv(6.0.1):c7804/90:delGen=80, _jo5r(6.0.1):c7330/119:delGen=93, _jp6z(6.0.1):c7377/43:delGen=36, _jq41(6.0.1):c6791/32:delGen=31, _jquf(6.0.1):c5259/7:delGen=7, _jtpr(6.0.1):c119, _jtox(6.0.1):c108, _jtod(6.0.1):c97, _jtph(6.0.1):c114, _jtq1(6.0.1):c117, _jto3(6.0.1):c98, _jtp7(6.0.1):c123, _jton(6.0.1):c118, _jtqa(6.0.1):C5, _jtqb(6.0.1):C5] packetCount=2
> 2016-09-02 12:49:11,810 level=WARN    thread=updateExecutor-2-thread-1-processing-http:////atlprwfftsol003:8983//solr//transcribedReports_shard3_replica3 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient Failed to parse error response from http://atlprwfftsol003:8983/solr/transcribedReports_shard3_replica3 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
> 2016-09-02 12:49:11,810 level=ERROR   thread=updateExecutor-2-thread-1-processing-http:////atlprwfftsol003:8983//solr//transcribedReports_shard3_replica3 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.update.StreamingSolrClients error
> org.apache.solr.common.SolrException: Unauthorized request, Response code: 401
>
>
>
> request: http://atlprwfftsol003:8983/solr/transcribedReports_shard3_replica3/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:11,810 level=WARN    thread=updateExecutor-2-thread-8-processing-http:////atlprwfftsol003:8983//solr//transcribedReports_shard4_replica4 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient Failed to parse error response from http://atlprwfftsol003:8983/solr/transcribedReports_shard4_replica4 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
> 2016-09-02 12:49:11,810 level=ERROR   thread=updateExecutor-2-thread-8-processing-http:////atlprwfftsol003:8983//solr//transcribedReports_shard4_replica4 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.update.StreamingSolrClients error
> org.apache.solr.common.SolrException: Unauthorized request, Response code: 401
>
>
>
> request: http://atlprwfftsol003:8983/solr/transcribedReports_shard4_replica4/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyTermDeletes took 1.7 msec for 25 segments and 1 packets; 5 del terms visited; 67 seg terms visited
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyDeletes: 0 new deleted documents
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyDeletes took 62 msec for 25 segments, 0 newly deleted docs (query deletes), 5 visited terms, allDeleted=null
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: prune sis=segments_htjo: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 minGen=183 packetCount=2
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: pruneDeletes: prune 2 packets; 0 packets remain
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: incRefDeleter for NRT reader version=3536052 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: return reader version=3536052 reader=StandardDirectoryReader(segments_htjo:3536052:nrt _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5)
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: autoExpireDocs-18-thread-1 finishFullFlush success=true
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]: findMerges: 25 segments
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_iin7(6.0.1):C491496/573:delGen=526 size=1181.402 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_fb82(6.0.1):C387586/788:delGen=727 size=933.087 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_5tw9(6.0.1):C241588/706:delGen=703 size=580.202 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_j7ia(6.0.1):c65875/307:delGen=272 size=162.085 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_iqm7(6.0.1):c67369/342:delGen=325 size=161.505 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jksn(6.0.1):c60239/75:delGen=70 size=144.827 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_iy15(6.0.1):c32975/162:delGen=157 size=78.643 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jthf(6.0.1):c27514/3:delGen=3 size=66.931 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jf5b(6.0.1):c22457/84:delGen=83 size=54.829 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_i0at(6.0.1):c17859/119:delGen=112 size=42.784 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jmdv(6.0.1):c7804/90:delGen=80 size=20.166 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jo5r(6.0.1):c7330/119:delGen=93 size=18.428 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jp6z(6.0.1):c7377/43:delGen=36 size=16.519 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jq41(6.0.1):c6791/32:delGen=31 size=15.395 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jquf(6.0.1):c5259/7:delGen=7 size=12.601 MB
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jton(6.0.1):c118 size=0.389 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtq1(6.0.1):c117 size=0.365 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtp7(6.0.1):c123 size=0.362 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtpr(6.0.1):c119 size=0.358 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtox(6.0.1):c108 size=0.335 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtph(6.0.1):c114 size=0.320 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jto3(6.0.1):c98 size=0.311 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtod(6.0.1):c97 size=0.287 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtqb(6.0.1):C5 size=0.021 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtqa(6.0.1):C5 size=0.019 MB [floored]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   allowedSegmentCount=31 vs count=25 (eligible count=25) tooBigCount=0
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][autoExpireDocs-18-thread-1]: now merge
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][autoExpireDocs-18-thread-1]:   index: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][autoExpireDocs-18-thread-1]:   no more merges pending; now return
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: getReader took 62 msec
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@7b51b06[transcribedReports_shard1_replica1] realtime]
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][autoExpireDocs-18-thread-1]: delete []
> 2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: decRefDeleter for NRT reader version=3536050 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5
> 2016-09-02 12:49:11,856 level=WARN    thread=updateExecutor-2-thread-7-processing-http:////atlprwfftsol002:8983//solr//transcribedReports_shard2_replica2 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient Failed to parse error response from http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
> 2016-09-02 12:49:11,856 level=ERROR   thread=updateExecutor-2-thread-7-processing-http:////atlprwfftsol002:8983//solr//transcribedReports_shard2_replica2 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.update.StreamingSolrClients error
> org.apache.solr.common.SolrException: Unauthorized request, Response code: 401
>
>
>
> request: http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:11,872 level=WARN    thread=updateExecutor-2-thread-1-processing-http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient Failed to parse error response from http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
> 2016-09-02 12:49:11,872 level=ERROR   thread=updateExecutor-2-thread-1-processing-http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.update.StreamingSolrClients error
> org.apache.solr.common.SolrException: Unauthorized request, Response code: 401
>
>
>
> request: http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/update?_version_=-1544364282917945344&update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:11,872 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
> 2016-09-02 12:49:11,872 level=ERROR   thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.DistributedUpdateProcessor Setting up to try to start recovery on replica http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/
> org.apache.solr.common.SolrException: Unauthorized request, Response code: 401
>
>
>
> request: http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/update?_version_=-1544364282917945344&update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
>         at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
>         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:11,872 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.cloud.ZkController Put replica core=transcribedReports_shard1_replica2 coreNodeName=core_node14 on atlprwfftsol002:8983_solr into leader-initiated recovery.
> 2016-09-02 12:49:11,872 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1] {deleteByQuery={!cache=false}expiryDate:[* TO 2016-09-02T12:49:11.794Z] (-1544364282917945344)} 0 82
> 2016-09-02 12:49:11,872 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.DocExpirationUpdateProcessorFactory Finished periodic deletion of expired docs
> 2016-09-02 12:49:11,872 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.common.cloud.SolrZkClient makePath: /collections/transcribedReports/leader_initiated_recovery/shard1
> 2016-09-02 12:49:11,903 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.ZkController Wrote down to /collections/transcribedReports/leader_initiated_recovery/shard1/core_node14
> 2016-09-02 12:49:11,903 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread Put replica core=transcribedReports_shard1_replica2 coreNodeName=core_node14 on atlprwfftsol002:8983_solr into leader-initiated recovery.
> 2016-09-02 12:49:11,903 level=WARN    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread Leader is publishing core=transcribedReports_shard1_replica2 coreNodeName =core_node14 state=down on behalf of un-reachable replica http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/
> 2016-09-02 12:49:11,935 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread LeaderInitiatedRecoveryThread-transcribedReports_shard1_replica2 started running to send REQUESTRECOVERY command to http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/; will try for a max of 600 secs
> 2016-09-02 12:49:11,935 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread Asking core=transcribedReports_shard1_replica2 coreNodeName=core_node14 on http://atlprwfftsol002:8983/solr to recover
> 2016-09-02 12:49:11,935 level=INFO    thread=zkCallback-4-thread-2-processing-n:atlprwfftsol001:8983_solr category=org.apache.solr.common.cloud.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/transcribedReports/state.json] for collection [transcribedReports] has occurred - updating... (live nodes size: [4])
> 2016-09-02 12:49:11,935 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread Successfully sent REQUESTRECOVERY command to core=transcribedReports_shard1_replica2 coreNodeName=core_node14 on http://atlprwfftsol002:8983/solr
> 2016-09-02 12:49:11,935 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread LeaderInitiatedRecoveryThread-transcribedReports_shard1_replica2 completed successfully after running for 74.0ms
> 2016-09-02 12:49:11,950 level=INFO    thread=zkCallback-4-thread-2-processing-n:atlprwfftsol001:8983_solr category=org.apache.solr.common.cloud.ZkStateReader Updating data for [transcribedReports] from [178079] to [178080]
> 2016-09-02 12:49:11,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation Going to wait for coreNodeName: core_node14, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
> 2016-09-02 12:49:11,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
> 2016-09-02 12:49:11,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation Will wait a max of 183 seconds to see transcribedReports_shard1_replica1 (shard1 of transcribedReports) have state: recovering
> 2016-09-02 12:49:11,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation In WaitForState(recovering): collection=transcribedReports, shard=shard1, thisCore=transcribedReports_shard1_replica1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=atlprwfftsol002:8983_solr, coreNodeName=core_node14, onlyIfActiveCheckResult=false, nodeProps: core_node14:{"state":"down","base_url":"http://atlprwfftsol002:8983/solr","core":"transcribedReports_shard1_replica2","node_name":"atlprwfftsol002:8983_solr"}
> 2016-09-02 12:49:12,091 level=INFO    thread=zkCallback-4-thread-2-processing-n:atlprwfftsol001:8983_solr category=org.apache.solr.common.cloud.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/transcribedReports/state.json] for collection [transcribedReports] has occurred - updating... (live nodes size: [4])
> 2016-09-02 12:49:12,091 level=INFO    thread=zkCallback-4-thread-2-processing-n:atlprwfftsol001:8983_solr category=org.apache.solr.common.cloud.ZkStateReader Updating data for [transcribedReports] from [178080] to [178081]
> 2016-09-02 12:49:12,310 level=INFO    thread=qtp762152757-20 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
> 2016-09-02 12:49:12,310 level=INFO    thread=qtp762152757-20 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
> 2016-09-02 12:49:12,310 level=INFO    thread=qtp762152757-20 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
> 2016-09-02 12:49:12,310 level=INFO    thread=qtp762152757-20 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
> 2016-09-02 12:49:12,325 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=7
> 2016-09-02 12:49:12,372 level=INFO    thread=qtp762152757-18 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50
> 2016-09-02 12:49:12,372 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2
> 2016-09-02 12:49:12,372 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
> java.io.IOException: Committed before 100 Continues
>         at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
>         at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
>         at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:12,419 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/&wt=javabin&version=2}{add=[711929582 (1544364283546042368)]} 0 6
> 2016-09-02 12:49:12,700 level=INFO    thread=qtp762152757-19 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
> 2016-09-02 12:49:12,700 level=INFO    thread=qtp762152757-19 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 1
> 2016-09-02 12:49:12,794 level=INFO    thread=qtp762152757-15 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol001:8983/solr/transcribedReports_shard2_replica1/&wt=javabin&version=2}{add=[711868996 (1544364283917238272)]} 0 55
> 2016-09-02 12:49:12,794 level=INFO    thread=qtp762152757-13 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711868996]} 0 68
> 2016-09-02 12:49:12,794 level=INFO    thread=qtp762152757-21 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/&wt=javabin&version=2}{add=[711816331 (1544364283949744128)]} 0 14
> 2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50
> 2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2
> 2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
> java.io.IOException: Committed before 100 Continues
>         at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
>         at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
>         at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
> 2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 1
> 2016-09-02 12:49:12,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation In WaitForState(recovering): collection=transcribedReports, shard=shard1, thisCore=transcribedReports_shard1_replica1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=atlprwfftsol002:8983_solr, coreNodeName=core_node14, onlyIfActiveCheckResult=false, nodeProps: core_node14:{"core":"transcribedReports_shard1_replica2","base_url":"http://atlprwfftsol002:8983/solr","node_name":"atlprwfftsol002:8983_solr","state":"recovering"}
> 2016-09-02 12:49:12,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation Waited coreNodeName: core_node14, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> 2016-09-02 12:49:12,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=atlprwfftsol002:8983_solr&onlyIfLeaderActive=true&core=transcribedReports_shard1_replica1&coreNodeName=core_node14&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=997
> 2016-09-02 12:49:13,216 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol004:8983/solr/transcribedReports_shard3_replica4/&wt=javabin&version=2}{add=[711817959 (1544364284392243200)]} 0 19
> 2016-09-02 12:49:13,497 level=INFO    thread=qtp762152757-13 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50
> 2016-09-02 12:49:13,497 level=INFO    thread=qtp762152757-13 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2
> 2016-09-02 12:49:13,497 level=INFO    thread=qtp762152757-13 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
> java.io.IOException: Committed before 100 Continues
>         at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
>         at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
>         at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:13,560 level=INFO    thread=qtp762152757-21 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/&wt=javabin&version=2}{add=[711926509 (1544364284725690368)]} 0 2
> 2016-09-02 12:49:13,638 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1472817956574&since=0} status=0 QTime=0
> 2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-20 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol001:8983/solr/transcribedReports_shard2_replica1/&wt=javabin&version=2}{add=[711929853 (1544364285015097344)]} 0 18
> 2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-12 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711929853]} 0 37
> 2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-17 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50
> 2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-17 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2
> 2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-17 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
> java.io.IOException: Committed before 100 Continues
>         at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
>         at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
>         at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
> 2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
> 2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
> 2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
> 2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=5
> 2016-09-02 12:49:17,310 level=INFO    thread=qtp762152757-19 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
> 2016-09-02 12:49:17,325 level=INFO    thread=qtp762152757-19 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
> 2016-09-02 12:49:17,325 level=INFO    thread=qtp762152757-19 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
> 2016-09-02 12:49:17,325 level=INFO    thread=qtp762152757-19 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
> 2016-09-02 12:49:17,325 level=INFO    thread=qtp762152757-19 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=6
> 2016-09-02 12:49:17,716 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
> 2016-09-02 12:49:17,716 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 3
> 2016-09-02 12:49:18,028 level=INFO    thread=qtp762152757-12 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
> 2016-09-02 12:49:18,028 level=INFO    thread=qtp762152757-12 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 1
> 2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-21 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol001:8983/solr/transcribedReports_shard2_replica1/&wt=javabin&version=2}{add=[711834143 (1544364289454768128)]} 0 19
> 2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-18 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711834143]} 0 41
> 2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-20 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50
> 2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2
> 2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
> java.io.IOException: Committed before 100 Continues
>         at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
>         at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
>         at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: start
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: enter lock
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: now prepare
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: prepareCommit: flush
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]:   index before flush _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: startFullFlush
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: anyChanges? numDocsInRam=8 deletes=true hasTickets:false pendingChangesInFullFlush: false
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWFC][commitScheduler-15-thread-1]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 7 deleted terms (unique count=7) bytesUsed=1211, segment=_i8a5, aborted=false, numDocsInRAM=8, deleteQueue=DWDQ: [ generation: 5 ]]
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flush postings as segment _i8a5 numDocs=8
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 1 msec to write norms
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 1 msec to write docValues
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 0 msec to write points
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 0 msec to finish stored fields
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 8 msec to write postings and finish vectors
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 0 msec to write fieldInfos
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: new segment has 0 deleted docs
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: new segment has no vectors; norms; docValues; prox; freqs
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flushedFiles=[_i8a5.nvd, _i8a5_Lucene50_0.tip, _i8a5_Lucene54_0.dvd, _i8a5_Lucene50_0.tim, _i8a5_Lucene54_0.dvm, _i8a5_Lucene50_0.doc, _i8a5.fdx, _i8a5.nvm, _i8a5_Lucene50_0.pos, _i8a5.fnm, _i8a5.fdt]
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flushed codec=Lucene60
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flushed: segment=_i8a5 ramUsed=0.282 MB newFlushedSize=0.033 MB docs/MB=243.064
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flush time 12.739399 msec
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: publishFlushedSegment seg-private updates=null
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: publishFlushedSegment
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: push deletes  8 deleted terms (unique count=8) bytesUsed=1072 segmentPrivate?=false delGen=13 packetCount=1 totBytesUsed=1072
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: publish sets newSegment delGen=14 seg=_i8a5(6.0.1):C8
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: now checkpoint "_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8" [26 segments ; isCommit = false]
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: delete []
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: 0 msec to checkpoint
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: apply all deletes during flush
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: now apply all deletes for all segments maxDoc=1450172
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: open segment readers took 0 msec
> 2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: infos=[_5uop(6.0.1):C241991/704:delGen=698, _fb71(6.0.1):C389680/682:delGen=628, _haro(6.0.1):C485464/597:delGen=546, _hewu(6.0.1):c27085/185:delGen=168, _hjdo(6.0.1):c77072/249:delGen=225, _hpme(6.0.1):c57230/309:delGen=297, _hw4u(6.0.1):c24068/137:delGen=130, _i154(6.0.1):c68871/95:delGen=93, _i6m2(6.0.1):c38230/11:delGen=11, _i1z4(6.0.1):c9132/67:delGen=66, _i2wq(6.0.1):c7487/87:delGen=81, _i3wa(6.0.1):c6191/132:delGen=104, _i4ry(6.0.1):c6282/62:delGen=56, _i6ls(6.0.1):c1356/7:delGen=7, _i7ui(6.0.1):c7820/6:delGen=6, _i898(6.0.1):c263, _i88e(6.0.1):c256, _i8a2(6.0.1):c306, _i86g(6.0.1):c275, _i89i(6.0.1):c273, _i89s(6.0.1):c307, _i88y(6.0.1):c261, _i88o(6.0.1):c261, _i8a3(6.0.1):C1, _i8a4(6.0.1):C2, _i8a5(6.0.1):C8] packetCount=1
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyTermDeletes took 0.9 msec for 26 segments and 1 packets; 8 del terms visited; 109 seg terms visited
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: 0 new deleted documents
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes took 15 msec for 26 segments, 0 newly deleted docs (query deletes), 8 visited terms, allDeleted=null
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: prune sis=segments_hidp: _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8 minGen=15 packetCount=1
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: pruneDeletes: prune 1 packets; 0 packets remain
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: commitScheduler-15-thread-1 finishFullFlush success=true
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]: findMerges: 26 segments
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_haro(6.0.1):C485464/597:delGen=546 size=1165.550 MB
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_fb71(6.0.1):C389680/682:delGen=628 size=938.388 MB
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_5uop(6.0.1):C241991/704:delGen=698 size=580.037 MB
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_hjdo(6.0.1):c77072/249:delGen=225 size=185.438 MB
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i154(6.0.1):c68871/95:delGen=93 size=165.278 MB
> 2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_hpme(6.0.1):c57230/309:delGen=297 size=137.766 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i6m2(6.0.1):c38230/11:delGen=11 size=90.215 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_hewu(6.0.1):c27085/185:delGen=168 size=63.326 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_hw4u(6.0.1):c24068/137:delGen=130 size=62.890 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i1z4(6.0.1):c9132/67:delGen=66 size=20.958 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i7ui(6.0.1):c7820/6:delGen=6 size=20.457 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i2wq(6.0.1):c7487/87:delGen=81 size=18.896 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i3wa(6.0.1):c6191/132:delGen=104 size=15.820 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i4ry(6.0.1):c6282/62:delGen=56 size=14.736 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i6ls(6.0.1):c1356/7:delGen=7 size=3.616 MB
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i89i(6.0.1):c273 size=0.820 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i8a2(6.0.1):c306 size=0.803 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i89s(6.0.1):c307 size=0.801 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i88y(6.0.1):c261 size=0.797 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i86g(6.0.1):c275 size=0.794 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i898(6.0.1):c263 size=0.769 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i88e(6.0.1):c256 size=0.761 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i88o(6.0.1):c261 size=0.736 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i8a5(6.0.1):C8 size=0.033 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i8a4(6.0.1):C2 size=0.019 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i8a3(6.0.1):C1 size=0.009 MB [floored]
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   allowedSegmentCount=31 vs count=26 (eligible count=26) tooBigCount=0
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-15-thread-1]: now merge
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-15-thread-1]:   index: _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-15-thread-1]:   no more merges pending; now return
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: startCommit(): start
> 2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: startCommit index=_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8 changeCount=16
> 2016-09-02 12:49:18,388 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: startCommit: wrote pending segments file "pending_segments_hidq"
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: start
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: enter lock
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: now prepare
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: prepareCommit: flush
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]:   index before flush _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: startFullFlush
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: anyChanges? numDocsInRam=5 deletes=true hasTickets:false pendingChangesInFullFlush: false
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWFC][commitScheduler-12-thread-1]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 4 deleted terms (unique count=4) bytesUsed=692, segment=_jtqc, aborted=false, numDocsInRAM=5, deleteQueue=DWDQ: [ generation: 92 ]]
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flush postings as segment _jtqc numDocs=5
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 0 msec to write norms
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 1 msec to write docValues
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 0 msec to write points
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 0 msec to finish stored fields
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 6 msec to write postings and finish vectors
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 0 msec to write fieldInfos
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: new segment has 0 deleted docs
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: new segment has no vectors; norms; docValues; prox; freqs
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flushedFiles=[_jtqc_Lucene50_0.doc, _jtqc_Lucene50_0.tim, _jtqc.fdt, _jtqc.nvm, _jtqc_Lucene50_0.tip, _jtqc.fnm, _jtqc_Lucene50_0.pos, _jtqc.fdx, _jtqc_Lucene54_0.dvm, _jtqc.nvd, _jtqc_Lucene54_0.dvd]
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flushed codec=Lucene60
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flushed: segment=_jtqc ramUsed=0.258 MB newFlushedSize=0.013 MB docs/MB=379.479
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flush time 10.267016 msec
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: publishFlushedSegment seg-private updates=null
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: publishFlushedSegment
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: push deletes  5 deleted terms (unique count=5) bytesUsed=1072 segmentPrivate?=false delGen=184 packetCount=1 totBytesUsed=1072
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: publish sets newSegment delGen=185 seg=_jtqc(6.0.1):C5
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5" [26 segments ; isCommit = false]
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: delete []
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: 0 msec to checkpoint
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: apply all deletes during flush
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: now apply all deletes for all segments maxDoc=1450428
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: open segment readers took 0 msec
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: infos=[_5tw9(6.0.1):C241588/706:delGen=703, _fb82(6.0.1):C387586/788:delGen=727, _iin7(6.0.1):C491496/573:delGen=526, _i0at(6.0.1):c17859/119:delGen=112, _iy15(6.0.1):c32975/162:delGen=157, _iqm7(6.0.1):c67369/342:delGen=325, _jf5b(6.0.1):c22457/84:delGen=83, _j7ia(6.0.1):c65875/307:delGen=272, _jksn(6.0.1):c60239/75:delGen=70, _jthf(6.0.1):c27514/3:delGen=3, _jmdv(6.0.1):c7804/90:delGen=80, _jo5r(6.0.1):c7330/119:delGen=93, _jp6z(6.0.1):c7377/43:delGen=36, _jq41(6.0.1):c6791/32:delGen=31, _jquf(6.0.1):c5259/7:delGen=7, _jtpr(6.0.1):c119, _jtox(6.0.1):c108, _jtod(6.0.1):c97, _jtph(6.0.1):c114, _jtq1(6.0.1):c117, _jto3(6.0.1):c98, _jtp7(6.0.1):c123, _jton(6.0.1):c118, _jtqa(6.0.1):C5, _jtqb(6.0.1):C5, _jtqc(6.0.1):C5] packetCount=1
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyTermDeletes took 1.1 msec for 26 segments and 1 packets; 5 del terms visited; 77 seg terms visited
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: 0 new deleted documents
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes took 0 msec for 26 segments, 0 newly deleted docs (query deletes), 5 visited terms, allDeleted=null
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: prune sis=segments_htjo: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 minGen=186 packetCount=1
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: pruneDeletes: prune 1 packets; 0 packets remain
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: commitScheduler-12-thread-1 finishFullFlush success=true
> 2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]: findMerges: 26 segments
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_iin7(6.0.1):C491496/573:delGen=526 size=1181.402 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_fb82(6.0.1):C387586/788:delGen=727 size=933.087 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_5tw9(6.0.1):C241588/706:delGen=703 size=580.202 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_j7ia(6.0.1):c65875/307:delGen=272 size=162.085 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_iqm7(6.0.1):c67369/342:delGen=325 size=161.505 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jksn(6.0.1):c60239/75:delGen=70 size=144.827 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_iy15(6.0.1):c32975/162:delGen=157 size=78.643 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jthf(6.0.1):c27514/3:delGen=3 size=66.931 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jf5b(6.0.1):c22457/84:delGen=83 size=54.829 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_i0at(6.0.1):c17859/119:delGen=112 size=42.784 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jmdv(6.0.1):c7804/90:delGen=80 size=20.166 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jo5r(6.0.1):c7330/119:delGen=93 size=18.428 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jp6z(6.0.1):c7377/43:delGen=36 size=16.519 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jq41(6.0.1):c6791/32:delGen=31 size=15.395 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jquf(6.0.1):c5259/7:delGen=7 size=12.601 MB
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jton(6.0.1):c118 size=0.389 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtq1(6.0.1):c117 size=0.365 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtp7(6.0.1):c123 size=0.362 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtpr(6.0.1):c119 size=0.358 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtox(6.0.1):c108 size=0.335 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtph(6.0.1):c114 size=0.320 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jto3(6.0.1):c98 size=0.311 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtod(6.0.1):c97 size=0.287 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtqb(6.0.1):C5 size=0.021 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtqa(6.0.1):C5 size=0.019 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtqc(6.0.1):C5 size=0.014 MB [floored]
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   allowedSegmentCount=31 vs count=26 (eligible count=26) tooBigCount=0
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-12-thread-1]: now merge
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-12-thread-1]:   index: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-12-thread-1]:   no more merges pending; now return
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: startCommit(): start
> 2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: startCommit index=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 changeCount=10
> 2016-09-02 12:49:18,528 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: startCommit: wrote pending segments file "pending_segments_htjp"
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: done all syncs: [_i6ls.cfe, _i8a3.fdt, _i86g.cfs, _i8a3.fdx, _i89i.si, _i3wa.cfs, _haro.fdt, _i86g.si, _i89s.cfe, _haro_Lucene50_0.tim, _i6ls.cfs, _i8a5.nvd, _haro_Lucene50_0.tip, _i8a5_Lucene54_0.dvd, _5uop_je.liv, _haro.si, _i8a5_Lucene54_0.dvm, _i3wa.cfe, _i8a5.nvm, _i8a5.fnm, _i6ls_7.liv, _i89s.cfs, _i8a3.nvd, _i4ry_1k.liv, _i8a5.fdx, _i4ry.si, _i8a5.fdt, _i88y.cfe, _hjdo.si, _i7ui_6.liv, _haro.fdx, _i1z4.cfs, _i88e.si, _i8a5_Lucene50_0.pos, _i88y.cfs, _i8a3_Lucene50_0.doc, _i8a3.nvm, _i8a3.fnm, _i8a5.si, _fb71.nvd, _hpme.si, _i154.si, _i1z4.cfe, _i6m2.si, _i88o.si, _hpme.cfe, _fb71.nvm, _fb71.fnm, _5uop.si, _i8a3_Lucene50_0.pos, _i6ls.si, _i8a2.si, _i7ui.cfs, _fb71.fdx, _haro_Lucene54_0.dvm, _hjdo.cfs, _i154_2l.liv, _i7ui.cfe, _hjdo.cfe, _haro_Lucene54_0.dvd, _i88e.cfs, _fb71_Lucene50_0.tip, _i2wq.si, _fb71_Lucene50_0.tim, _i8a5_Lucene50_0.tip, _i8a5_Lucene50_0.tim, _i89s.si, _i8a2.cfs, _i898.si, _i88e.cfe, _hewu.si, _fb71_Lucene54_0.dvm, _i4ry.cfe, _i8a2.cfe, _i8a4_Lucene50_0.doc, _hw4u.si, _5uop_Lucene50_0.pos, _fb71_Lucene54_0.dvd, _i88y.si, _i8a3_Lucene50_0.tip, _fb71_hg.liv, _i88o.cfs, _i8a4.fdx, _fb71.fdt, _i8a3_Lucene50_0.tim, _i4ry.cfs, _i8a4.fdt, _i8a4.si, _5uop_Lucene54_0.dvd, _hewu.cfe, _i898.cfs, _5uop.nvd, _i8a4_Lucene50_0.pos, _5uop_Lucene50_0.doc, _5uop_Lucene54_0.dvm, _i898.cfe, _hewu_4o.liv, _i2wq_29.liv, _i8a4.nvm, _i8a4.fnm, _i3wa.si, _5uop.nvm, _5uop.fnm, _hewu.cfs, _haro_f6.liv, _i8a3_Lucene54_0.dvd, _i8a4.nvd, _i8a3_Lucene54_0.dvm, _hpme.cfs, _i3wa_2w.liv, _i2wq.cfe, _hw4u_3m.liv, _hpme_89.liv, _5uop_Lucene50_0.tip, _i8a4_Lucene54_0.dvd, _fb71_Lucene50_0.pos, _i1z4.si, _hjdo_69.liv, _5uop.fdx, _5uop_Lucene50_0.tim, _fb71.si, _hw4u.cfe, _i6m2.cfe, _i8a4_Lucene54_0.dvm, _5uop.fdt, _fb71_Lucene50_0.doc, _haro.nvd, _hw4u.cfs, _i8a5_Lucene50_0.doc, _haro.fnm, _haro.nvm, _i6m2.cfs, _i154.cfs, _i89i.cfs, _i8a4_Lucene50_0.tip, _i6m2_b.liv, _i88o.cfe, _haro_Lucene50_0.pos, _i89i.cfe, _i8a4_Lucene50_0.tim, _i154.cfe, _haro_Lucene50_0.doc, _i2wq.cfs, _i86g.cfe, _i8a3.si, _i1z4_1u.liv, _i7ui.si]
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: pendingCommit != null
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: done writing segments file "segments_hidq"
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: now checkpoint "_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8" [26 segments ; isCommit = true]
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.core.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
>         commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard2_replica1\data\index.20160823134952477 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_hidp,generation=816973}
>         commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard2_replica1\data\index.20160823134952477 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_hidq,generation=816974}
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.core.SolrDeletionPolicy newest commit generation = 816974
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: deleteCommits: now decRef commit "segments_hidp"
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: delete [segments_hidp]
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: 4 msec to checkpoint
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: delete []
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: took 388.6 msec
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: done
> 2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: nrtIsCurrent: infoVersion matches: false; DW changes: false; BD changes: false
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: flush at getReader
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: startFullFlush
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: apply all deletes during flush
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: now apply all deletes for all segments maxDoc=1450172
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: open segment readers took 0 msec
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: no segments; skipping
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: prune sis=segments_hidq: _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8 minGen=15 packetCount=0
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: incRefDeleter for NRT reader version=3357331 segments=_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: return reader version=3357331 reader=StandardDirectoryReader(segments_hidq:3357331:nrt _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8)
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: commitScheduler-15-thread-1 finishFullFlush success=true
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: getReader took 15 msec
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@4320bac9[transcribedReports_shard2_replica1] realtime]
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: delete []
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: decRefDeleter for NRT reader version=3357327 segments=_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2
> 2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.DirectUpdateHandler2 end_commit_flush
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: done all syncs: [_jtqb.fdt, _jto3.cfe, _jksn.cfs, _iy15.cfs, _jtod.cfe, _jtqb.fdx, _jtqa_Lucene50_0.tim, _fb82.fdx, _jtox.cfe, _jksn_1y.liv, _jmdv.si, _iy15_4d.liv, _jto3.cfs, _iqm7.cfs, _5tw9_Lucene50_0.pos, _iin7.fdt, _jksn.cfe, _jq41.cfe, _jtqb_Lucene50_0.pos, _jtqa_Lucene54_0.dvm, _iin7.fdx, _jf5b.cfe, _iin7_Lucene50_0.pos, _iy15.cfe, _jf5b.si, _iqm7.si, _jtq1.cfs, _iqm7.cfe, _jtox.cfs, _jtox.si, _fb82_Lucene54_0.dvm, _jf5b.cfs, _jq41.cfs, _fb82_Lucene54_0.dvd, _jtqb.nvd, _fb82_Lucene50_0.pos, _iin7.si, _jf5b_2b.liv, _fb82_Lucene50_0.doc, _jtqb.si, _jtqb.nvm, _jtqb.fnm, _5tw9_Lucene50_0.doc, _iin7.nvd, _jq41_v.liv, _fb82.fdt, _iin7.nvm, _iin7.fnm, _jtph.cfs, _jquf_7.liv, _jthf.si, _5tw9_Lucene54_0.dvm, _jthf_3.liv, _jtpr.si, _5tw9_Lucene54_0.dvd, _fb82_Lucene50_0.tim, _5tw9_Lucene50_0.tim, _jtqc.si, _fb82_Lucene50_0.tip, _5tw9_Lucene50_0.tip, _i0at.si, _jton.si, _fb82.nvd, _jquf.cfs, _fb82.nvm, _fb82.fnm, _iin7_em.liv, _jtqa_Lucene54_0.dvd, _iin7_Lucene50_0.doc, _j7ia.si, _iqm7_91.liv, _jtph.cfe, _jquf.cfe, _jtqb_Lucene50_0.doc, _jtqc_Lucene54_0.dvm, _jtqa_Lucene50_0.tip, _jton.cfe, _jo5r.cfs, _fb82_k7.liv, _5tw9.fdt, _jthf.cfs, _jton.cfs, _j7ia.cfe, _jmdv.cfe, _5tw9.fdx, _i0at_34.liv, _jtp7.si, _jtqc_Lucene54_0.dvd, _jp6z.si, _jtqa.si, _jtqc.fdt, _j7ia_7k.liv, _jmdv.cfs, _jtqc.fdx, _jto3.si, _jtqc_Lucene50_0.tim, _jtqc.nvm, _jtqc_Lucene50_0.tip, _jtqc.fnm, _iy15.si, _5tw9.si, _jtqc.nvd, _jthf.cfe, _jtqc_Lucene50_0.doc, _jtp7.cfs, _jtqa_Lucene50_0.pos, _jtqa.nvm, _jtqa.fnm, _i0at.cfe, _jtqb_Lucene54_0.dvm, _jmdv_28.liv, _jo5r.si, _iin7_Lucene50_0.tip, _i0at.cfs, _jo5r_2l.liv, _iin7_Lucene50_0.tim, _jp6z.cfe, _jtqa.nvd, _jtp7.cfe, _5tw9_jj.liv, _jtqb_Lucene50_0.tim, _jtpr.cfs, _iin7_Lucene54_0.dvm, _jp6z.cfs, _jtqc_Lucene50_0.pos, _jtod.si, _jtqa_Lucene50_0.doc, _jp6z_10.liv, _jq41.si, _jtqb_Lucene50_0.tip, _jksn.si, _jtq1.cfe, _5tw9.nvd, _jtpr.cfe, _jtqa.fdx, _jtph.si, _j7ia.cfs, _jtqa.fdt, _jo5r.cfe, _jtod.cfs, _5tw9.nvm, _5tw9.fnm, _jtqb_Lucene54_0.dvd, _jquf.si, _iin7_Lucene54_0.dvd, _fb82.si, _jtq1.si]
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: pendingCommit != null
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: done writing segments file "segments_htjp"
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5" [26 segments ; isCommit = true]
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.core.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
>         commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard1_replica1\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_htjo,generation=831444}
>         commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard1_replica1\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_htjp,generation=831445}
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.core.SolrDeletionPolicy newest commit generation = 831445
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: deleteCommits: now decRef commit "segments_htjo"
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: delete [segments_htjo]
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: 0 msec to checkpoint
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: delete []
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: took 435.8 msec
> 2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: done
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: nrtIsCurrent: infoVersion matches: false; DW changes: false; BD changes: false
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: flush at getReader
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: startFullFlush
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: apply all deletes during flush
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: now apply all deletes for all segments maxDoc=1450428
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: open segment readers took 0 msec
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: no segments; skipping
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: prune sis=segments_htjp: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 minGen=186 packetCount=0
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: incRefDeleter for NRT reader version=3536056 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: return reader version=3536056 reader=StandardDirectoryReader(segments_htjp:3536056:nrt _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5)
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: commitScheduler-12-thread-1 finishFullFlush success=true
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: getReader took 0 msec
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@3b4ec22[transcribedReports_shard1_replica1] realtime]
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: delete []
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: decRefDeleter for NRT reader version=3536052 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5
> 2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.DirectUpdateHandler2 end_commit_flush
> 2016-09-02 12:49:19,106 level=INFO    thread=qtp762152757-19 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50
> 2016-09-02 12:49:19,106 level=INFO    thread=qtp762152757-19 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2
> 2016-09-02 12:49:19,106 level=INFO    thread=qtp762152757-19 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
> java.io.IOException: Committed before 100 Continues
>         at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
>         at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
>         at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:19,153 level=INFO    thread=qtp762152757-18 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711925396]} 0 37
> 2016-09-02 12:49:19,169 level=INFO    thread=qtp762152757-20 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50
> 2016-09-02 12:49:19,169 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2
> 2016-09-02 12:49:19,169 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
> java.io.IOException: Committed before 100 Continues
>         at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
>         at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
>         at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:19,919 level=INFO    thread=qtp762152757-20 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol004:8983/solr/transcribedReports_shard3_replica4/&wt=javabin&version=2}{add=[711869839 (1544364291420848128)]} 0 17
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: anyChanges? numDocsInRam=1 deletes=true hasTickets:false pendingChangesInFullFlush: false
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: nrtIsCurrent: infoVersion matches: false; DW changes: true; BD changes: false
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: flush at getReader
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: startFullFlush
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: anyChanges? numDocsInRam=1 deletes=true hasTickets:false pendingChangesInFullFlush: false
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWFC][qtp762152757-17]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_jtqd, aborted=false, numDocsInRAM=1, deleteQueue=DWDQ: [ generation: 94 ]]
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flush postings as segment _jtqd numDocs=1
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 1 msec to write norms
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 0 msec to write docValues
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 0 msec to write points
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 2 msec to finish stored fields
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 2 msec to write postings and finish vectors
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 0 msec to write fieldInfos
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: new segment has 0 deleted docs
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: new segment has no vectors; norms; docValues; prox; freqs
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flushedFiles=[_jtqd_Lucene50_0.tip, _jtqd.nvm, _jtqd_Lucene54_0.dvm, _jtqd_Lucene50_0.tim, _jtqd_Lucene50_0.doc, _jtqd.fdx, _jtqd_Lucene50_0.pos, _jtqd.nvd, _jtqd.fdt, _jtqd.fnm, _jtqd_Lucene54_0.dvd]
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flushed codec=Lucene60
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flushed: segment=_jtqd ramUsed=0.242 MB newFlushedSize=0.004 MB docs/MB=253.402
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flush time 8.600604 msec
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: publishFlushedSegment seg-private updates=null
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: publishFlushedSegment
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: push deletes  1 deleted terms (unique count=1) bytesUsed=1072 segmentPrivate?=false delGen=188 packetCount=1 totBytesUsed=1072
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: publish sets newSegment delGen=189 seg=_jtqd(6.0.1):C1
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1" [27 segments ; isCommit = false]
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete []
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: 0 msec to checkpoint
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: apply all deletes during flush
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: now apply all deletes for all segments maxDoc=1450429
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: open segment readers took 0 msec
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: infos=[_5tw9(6.0.1):C241588/706:delGen=703, _fb82(6.0.1):C387586/788:delGen=727, _iin7(6.0.1):C491496/573:delGen=526, _i0at(6.0.1):c17859/119:delGen=112, _iy15(6.0.1):c32975/162:delGen=157, _iqm7(6.0.1):c67369/342:delGen=325, _jf5b(6.0.1):c22457/84:delGen=83, _j7ia(6.0.1):c65875/307:delGen=272, _jksn(6.0.1):c60239/75:delGen=70, _jthf(6.0.1):c27514/3:delGen=3, _jmdv(6.0.1):c7804/90:delGen=80, _jo5r(6.0.1):c7330/119:delGen=93, _jp6z(6.0.1):c7377/43:delGen=36, _jq41(6.0.1):c6791/32:delGen=31, _jquf(6.0.1):c5259/7:delGen=7, _jtpr(6.0.1):c119, _jtox(6.0.1):c108, _jtod(6.0.1):c97, _jtph(6.0.1):c114, _jtq1(6.0.1):c117, _jto3(6.0.1):c98, _jtp7(6.0.1):c123, _jton(6.0.1):c118, _jtqa(6.0.1):C5, _jtqb(6.0.1):C5, _jtqc(6.0.1):C5, _jtqd(6.0.1):C1] packetCount=1
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyTermDeletes took 1.1 msec for 27 segments and 1 packets; 1 del terms visited; 28 seg terms visited
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: 0 new deleted documents
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes took 0 msec for 27 segments, 0 newly deleted docs (query deletes), 1 visited terms, allDeleted=null
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: prune sis=segments_htjp: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1 minGen=190 packetCount=1
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: pruneDeletes: prune 1 packets; 0 packets remain
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: incRefDeleter for NRT reader version=3536058 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: return reader version=3536058 reader=StandardDirectoryReader(segments_htjp:3536058:nrt _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1)
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: qtp762152757-17 finishFullFlush success=true
> 2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]: findMerges: 27 segments
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_iin7(6.0.1):C491496/573:delGen=526 size=1181.402 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_fb82(6.0.1):C387586/788:delGen=727 size=933.087 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_5tw9(6.0.1):C241588/706:delGen=703 size=580.202 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_j7ia(6.0.1):c65875/307:delGen=272 size=162.085 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_iqm7(6.0.1):c67369/342:delGen=325 size=161.505 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jksn(6.0.1):c60239/75:delGen=70 size=144.827 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_iy15(6.0.1):c32975/162:delGen=157 size=78.643 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jthf(6.0.1):c27514/3:delGen=3 size=66.931 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jf5b(6.0.1):c22457/84:delGen=83 size=54.829 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_i0at(6.0.1):c17859/119:delGen=112 size=42.784 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jmdv(6.0.1):c7804/90:delGen=80 size=20.166 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jo5r(6.0.1):c7330/119:delGen=93 size=18.428 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jp6z(6.0.1):c7377/43:delGen=36 size=16.519 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jq41(6.0.1):c6791/32:delGen=31 size=15.395 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jquf(6.0.1):c5259/7:delGen=7 size=12.601 MB
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jton(6.0.1):c118 size=0.389 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtq1(6.0.1):c117 size=0.365 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtp7(6.0.1):c123 size=0.362 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtpr(6.0.1):c119 size=0.358 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtox(6.0.1):c108 size=0.335 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtph(6.0.1):c114 size=0.320 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jto3(6.0.1):c98 size=0.311 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtod(6.0.1):c97 size=0.287 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtqb(6.0.1):C5 size=0.021 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtqa(6.0.1):C5 size=0.019 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtqc(6.0.1):C5 size=0.014 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtqd(6.0.1):C1 size=0.004 MB [floored]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   allowedSegmentCount=31 vs count=27 (eligible count=27) tooBigCount=0
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [MS][qtp762152757-17]: now merge
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [MS][qtp762152757-17]:   index: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [MS][qtp762152757-17]:   no more merges pending; now return
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: getReader took 16 msec
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@4bffc6cf[transcribedReports_shard1_replica1] realtime]
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete []
> 2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: decRefDeleter for NRT reader version=3536056 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5
> 2016-09-02 12:49:20,060 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.IndexFingerprint IndexFingerprint millis:42.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=1544364291420848128, maxInHash=1544364291420848128, versionsHash=5743601267512047892, numVersions=1446979, numDocs=1446979, maxDoc=1450429}
> 2016-09-02 12:49:20,060 level=INFO    thread=qtp762152757-17 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=true&getVersions=100&wt=javabin&version=2} status=0 QTime=65
> 2016-09-02 12:49:20,060 level=INFO    thread=qtp762152757-16 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/&wt=javabin&version=2}{add=[711880061 (1544364291558211584)]} 0 7
> 2016-09-02 12:49:20,075 level=INFO    thread=qtp762152757-19 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/get params={getUpdates=-1544364282917945344&distrib=false&qt=/get&onlyIfActive=false&wt=javabin&version=2} status=0 QTime=1
> 2016-09-02 12:49:20,075 level=INFO    thread=qtp762152757-19 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
> 2016-09-02 12:49:20,075 level=INFO    thread=qtp762152757-19 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 2
> 2016-09-02 12:49:20,106 level=INFO    thread=qtp762152757-13 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711880746]} 0 58
> 2016-09-02 12:49:20,106 level=INFO    thread=qtp762152757-21 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50
> 2016-09-02 12:49:20,106 level=INFO    thread=qtp762152757-21 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2
> 2016-09-02 12:49:20,106 level=INFO    thread=qtp762152757-21 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
> java.io.IOException: Committed before 100 Continues
>         at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
>         at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
>         at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
>         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
>         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
>         at java.lang.Thread.run(Unknown Source)
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: start
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: enter lock
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: now prepare
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: prepareCommit: flush
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]:   index before flush _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: startFullFlush
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: apply all deletes during flush
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: now apply all deletes for all segments maxDoc=1450429
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: open segment readers took 0 msec
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: no segments; skipping
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: prune sis=segments_htjp: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1 minGen=190 packetCount=0
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: qtp762152757-17 finishFullFlush success=true
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: startCommit(): start
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: startCommit index=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1 changeCount=14
> 2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: startCommit: wrote pending segments file "pending_segments_htjq"
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: done all syncs: [_jtqb.fdt, _jto3.cfe, _jksn.cfs, _iy15.cfs, _jtod.cfe, _jtqb.fdx, _jtqa_Lucene50_0.tim, _fb82.fdx, _jtox.cfe, _jtqd.nvd, _jksn_1y.liv, _jmdv.si, _iy15_4d.liv, _jto3.cfs, _iqm7.cfs, _5tw9_Lucene50_0.pos, _iin7.fdt, _jksn.cfe, _jq41.cfe, _jtqb_Lucene50_0.pos, _jtqd.nvm, _jtqa_Lucene54_0.dvm, _iin7.fdx, _jf5b.cfe, _iin7_Lucene50_0.pos, _iy15.cfe, _jf5b.si, _jtqd.fdx, _iqm7.si, _jtq1.cfs, _iqm7.cfe, _jtox.cfs, _jtox.si, _jtqd.fdt, _fb82_Lucene54_0.dvm, _jf5b.cfs, _jq41.cfs, _fb82_Lucene54_0.dvd, _jtqb.nvd, _fb82_Lucene50_0.pos, _jtqd.fnm, _iin7.si, _jf5b_2b.liv, _fb82_Lucene50_0.doc, _jtqd_Lucene50_0.tip, _jtqb.si, _jtqb.nvm, _jtqb.fnm, _5tw9_Lucene50_0.doc, _jtqd_Lucene50_0.tim, _iin7.nvd, _jq41_v.liv, _fb82.fdt, _iin7.nvm, _iin7.fnm, _jtph.cfs, _jquf_7.liv, _jthf.si, _5tw9_Lucene54_0.dvm, _jthf_3.liv, _jtpr.si, _5tw9_Lucene54_0.dvd, _fb82_Lucene50_0.tim, _5tw9_Lucene50_0.tim, _jtqc.si, _fb82_Lucene50_0.tip, _5tw9_Lucene50_0.tip, _i0at.si, _jton.si, _fb82.nvd, _jquf.cfs, _fb82.nvm, _fb82.fnm, _iin7_em.liv, _jtqa_Lucene54_0.dvd, _iin7_Lucene50_0.doc, _j7ia.si, _iqm7_91.liv, _jtph.cfe, _jquf.cfe, _jtqb_Lucene50_0.doc, _jtqc_Lucene54_0.dvm, _jtqa_Lucene50_0.tip, _jtqd_Lucene50_0.pos, _jton.cfe, _jo5r.cfs, _fb82_k7.liv, _5tw9.fdt, _jthf.cfs, _jton.cfs, _j7ia.cfe, _jmdv.cfe, _5tw9.fdx, _i0at_34.liv, _jtp7.si, _jtqc_Lucene54_0.dvd, _jp6z.si, _jtqa.si, _jtqc.fdt, _j7ia_7k.liv, _jmdv.cfs, _jtqc.fdx, _jtqd_Lucene50_0.doc, _jto3.si, _jtqc_Lucene50_0.tim, _jtqc.nvm, _jtqc_Lucene50_0.tip, _jtqc.fnm, _iy15.si, _5tw9.si, _jtqc.nvd, _jthf.cfe, _jtqc_Lucene50_0.doc, _jtp7.cfs, _jtqa_Lucene50_0.pos, _jtqa.nvm, _jtqa.fnm, _i0at.cfe, _jtqb_Lucene54_0.dvm, _jmdv_28.liv, _jo5r.si, _iin7_Lucene50_0.tip, _i0at.cfs, _jo5r_2l.liv, _iin7_Lucene50_0.tim, _jp6z.cfe, _jtqa.nvd, _jtp7.cfe, _5tw9_jj.liv, _jtqb_Lucene50_0.tim, _jtpr.cfs, _jtqd_Lucene54_0.dvd, _iin7_Lucene54_0.dvm, _jp6z.cfs, _jtqc_Lucene50_0.pos, _jtqd_Lucene54_0.dvm, _jtod.si, _jtqa_Lucene50_0.doc, _jp6z_10.liv, _jq41.si, _jtqb_Lucene50_0.tip, _jksn.si, _jtq1.cfe, _5tw9.nvd, _jtpr.cfe, _jtqa.fdx, _jtph.si, _j7ia.cfs, _jtqa.fdt, _jtqd.si, _jo5r.cfe, _jtod.cfs, _5tw9.nvm, _5tw9.fnm, _jtqb_Lucene54_0.dvd, _jquf.si, _iin7_Lucene54_0.dvd, _fb82.si, _jtq1.si]
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: pendingCommit != null
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: done writing segments file "segments_htjq"
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1" [27 segments ; isCommit = true]
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.core.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
>         commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard1_replica1\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_htjp,generation=831445}
>         commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard1_replica1\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_htjq,generation=831446}
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.core.SolrDeletionPolicy newest commit generation = 831446
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: deleteCommits: now decRef commit "segments_htjp"
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete [segments_htjp]
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: 0 msec to checkpoint
> 2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete []
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: took 211.5 msec
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: done
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: nrtIsCurrent: infoVersion matches: false; DW changes: false; BD changes: false
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: flush at getReader
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: startFullFlush
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: apply all deletes during flush
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: now apply all deletes for all segments maxDoc=1450429
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: open segment readers took 0 msec
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: no segments; skipping
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: prune sis=segments_htjq: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1 minGen=190 packetCount=0
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: incRefDeleter for NRT reader version=3536060 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: return reader version=3536060 reader=StandardDirectoryReader(segments_htjq:3536060:nrt _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1)
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: qtp762152757-17 finishFullFlush success=true
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: getReader took 0 msec
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@11297ec7[transcribedReports_shard1_replica1] realtime]
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete []
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: decRefDeleter for NRT reader version=3536058 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.DirectUpdateHandler2 end_commit_flush
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&commit_end_point=true&wt=javabin&version=2}{commit=} 0 217
> 2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-19 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
> 2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
> 2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
> 2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
> 2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
> 2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=6
> 2016-09-02 12:49:20,450 level=INFO    thread=qtp762152757-12 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/replication params={generation=831446&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=57
> 2016-09-02 12:49:20,544 level=INFO    thread=qtp762152757-15 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/replication params={generation=831446&qt=/replication&file=_5tw9_Lucene54_0.dvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
> 2016-09-02 12:49:20,560 level=INFO    thread=qtp762152757-17 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/replication params={generation=831446&qt=/replication&file=_5tw9.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
> 2016-09-02 12:49:20,669 level=INFO    thread=qtp762152757-16 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol003:8983/solr/transcribedReports_shard3_replica3/&wt=javabin&version=2}{add=[711926412 (1544364292191551488)]} 0 33
> 2016-09-02 12:49:20,685 level=INFO    thread=qtp762152757-13 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
> 2016-09-02 12:49:20,685 level=INFO    thread=qtp762152757-13 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 2
>
> 2/ clusterstate.json
>
>     "solr_cluster_status.json.cluster": {
>         "collections": {
>             "transcribedReports": {
>                 "autoAddReplicas": "false",
>                 "configName": "transcribedReports",
>                 "maxShardsPerNode": "4",
>                 "replicationFactor": "3",
>                 "router": {
>                     "name": "compositeId"
>                 },
>                 "shards": {
>                     "shard1": {
>                         "range": "80000000-bfffffff",
>                         "replicas": {
>                             "core_node13": {
>                                 "base_url": "http://atlprwfftsol001:8983/solr",
>                                 "core": "transcribedReports_shard1_replica1",
>                                 "leader": "true",
>                                 "node_name": "atlprwfftsol001:8983_solr",
>                                 "state": "active"
>                             },
>                             "core_node14": {
>                                 "base_url": "http://atlprwfftsol002:8983/solr",
>                                 "core": "transcribedReports_shard1_replica2",
>                                 "node_name": "atlprwfftsol002:8983_solr",
>                                 "state": "down"
>                             }
>                         },
>                         "state": "active"
>                     },
>                     "shard2": {
>                         "range": "c0000000-ffffffff",
>                         "replicas": {
>                             "core_node4": {
>                                 "base_url": "http://atlprwfftsol001:8983/solr",
>                                 "core": "transcribedReports_shard2_replica1",
>                                 "node_name": "atlprwfftsol001:8983_solr",
>                                 "state": "active"
>                             },
>                             "core_node8": {
>                                 "base_url": "http://atlprwfftsol002:8983/solr",
>                                 "core": "transcribedReports_shard2_replica2",
>                                 "leader": "true",
>                                 "node_name": "atlprwfftsol002:8983_solr",
>                                 "state": "active"
>                             }
>                         },
>                         "state": "active"
>                     },
>                     "shard3": {
>                         "range": "0-3fffffff",
>                         "replicas": {
>                             "core_node10": {
>                                 "base_url": "http://atlprwfftsol003:8983/solr",
>                                 "core": "transcribedReports_shard3_replica3",
>                                 "leader": "true",
>                                 "node_name": "atlprwfftsol003:8983_solr",
>                                 "state": "active"
>                             },
>                             "core_node12": {
>                                 "base_url": "http://atlprwfftsol004:8983/solr",
>                                 "core": "transcribedReports_shard3_replica4",
>                                 "node_name": "atlprwfftsol004:8983_solr",
>                                 "state": "active"
>                             }
>                         },
>                         "state": "active"
>                     },
>                     "shard4": {
>                         "range": "40000000-7fffffff",
>                         "replicas": {
>                             "core_node11": {
>                                 "base_url": "http://atlprwfftsol003:8983/solr",
>                                 "core": "transcribedReports_shard4_replica4",
>                                 "leader": "true",
>                                 "node_name": "atlprwfftsol003:8983_solr",
>                                 "state": "active"
>                             },
>                             "core_node9": {
>                                 "base_url": "http://atlprwfftsol004:8983/solr",
>                                 "core": "transcribedReports_shard4_replica3",
>                                 "node_name": "atlprwfftsol004:8983_solr",
>                                 "state": "active"
>                             }
>                         },
>                         "state": "active"
>                     }
>                 },
>                 "znodeVersion": 178208
>             }
>         },
>         "live_nodes": [
>             "atlprwfftsol002:8983_solr",
>             "atlprwfftsol004:8983_solr",
>             "atlprwfftsol003:8983_solr",
>             "atlprwfftsol001:8983_solr"
>         ]
>     }
>
>
> 3/ transcribedReports cluster's solrconfig.xml.  I think the only changes to default are for the updateRequestProcessorChain (ignore commits from clients and auto delete expired documents), and auto commit configuration hard autocommit 15 seconds, opensearch=false soft autocommit every 15 *minutes*)
>
>     <updateRequestProcessorChain name="FFTtranscribedReportsProcessorChain" default="true">
>         <processor class="solr.IgnoreCommitOptimizeUpdateProcessorFactory">
>           <int name="statusCode">200</int>
>         </processor>
>         <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
>             <int name="autoDeletePeriodSeconds">30</int>
>             <str name="ttlFieldName">_ttl_</str>
>             <str name="expirationFieldName">expiryDate</str>
>         </processor>
>         <processor class="solr.FirstFieldValueUpdateProcessorFactory">
>             <str name="fieldName">expiryDate</str>
>         </processor>
>         <processor class="solr.LogUpdateProcessorFactory" />
>         <processor class="solr.DistributedUpdateProcessorFactory" />
>         <processor class="solr.RunUpdateProcessorFactory" />
>     </updateRequestProcessorChain>
>
>
>         <autoCommit>
>             <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>             <openSearcher>false</openSearcher>
>         </autoCommit>
>
>         <!-- setting softAutoCommit to 15 minutes for fft transcribedReports usage. -->
>         <autoSoftCommit>
>             <maxTime>${solr.autoSoftCommit.maxTime:900000}</maxTime>
>         </autoSoftCommit>
>
> (full solrconfig.xml for this collection below)
>
>
> <?xml version="1.0" encoding="UTF-8" ?>
> <!--
>  Licensed to the Apache Software Foundation (ASF) under one or more
>  contributor license agreements.  See the NOTICE file distributed with
>  this work for additional information regarding copyright ownership.
>  The ASF licenses this file to You under the Apache License, Version 2.0
>  (the "License"); you may not use this file except in compliance with
>  the License.  You may obtain a copy of the License at
>
>      http://www.apache.org/licenses/LICENSE-2.0
>
>  Unless required by applicable law or agreed to in writing, software
>  distributed under the License is distributed on an "AS IS" BASIS,
>  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>  See the License for the specific language governing permissions and
>  limitations under the License.
> -->
>
> <!--
>      For more details about configurations options that may appear in
>      this file, see http://wiki.apache.org/solr/SolrConfigXml.
> -->
> <config>
>
>     <!-- Having this in the config means that Solr will honour schema.xml rather
>          than constructing its own schema (which it calls "managed-schema")
>          This means we can manage the schema using the zkCli script in solr and
>          keep all the config under version control.
>     -->
>     <schemaFactory class="ClassicIndexSchemaFactory"/>
>
>     <!-- In all configuration below, a prefix of "solr." for class names
>          is an alias that causes solr to search appropriate packages,
>          including org.apache.solr.(search|update|request|core|analysis)
>
>          You may also specify a fully qualified Java classname if you
>          have your own custom plugins.
>       -->
>
>     <!-- Controls what version of Lucene various components of Solr
>          adhere to.  Generally, you want to use the latest version to
>          get all bug fixes and improvements. It is highly recommended
>          that you fully re-index after changing this setting as it can
>          affect both how text is indexed and queried.
>     -->
>     <luceneMatchVersion>6.0.0</luceneMatchVersion>
>
>     <!-- Data Directory
>
>          Used to specify an alternate directory to hold all index data
>          other than the default ./data under the Solr home.  If
>          replication is in use, this should match the replication
>          configuration.
>       -->
>     <dataDir>${solr.data.dir:}</dataDir>
>
>
>     <!-- The DirectoryFactory to use for indexes.
>
>          solr.StandardDirectoryFactory is filesystem
>          based and tries to pick the best implementation for the current
>          JVM and platform.  solr.NRTCachingDirectoryFactory, the default,
>          wraps solr.StandardDirectoryFactory and caches small files in memory
>          for better NRT performance.
>
>          One can force a particular implementation via solr.MMapDirectoryFactory,
>          solr.NIOFSDirectoryFactory, or solr.SimpleFSDirectoryFactory.
>
>          solr.RAMDirectoryFactory is memory based, not
>          persistent, and doesn't work with replication.
>       -->
>     <directoryFactory name="DirectoryFactory"
>                       class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}">
>     </directoryFactory>
>
>     <!-- The CodecFactory for defining the format of the inverted index.
>          The default implementation is SchemaCodecFactory, which is the official Lucene
>          index format, but hooks into the schema to provide per-field customization of
>          the postings lists and per-document values in the fieldType element
>          (postingsFormat/docValuesFormat). Note that most of the alternative implementations
>          are experimental, so if you choose to customize the index format, it's a good
>          idea to convert back to the official format e.g. via IndexWriter.addIndexes(IndexReader)
>          before upgrading to a newer version to avoid unnecessary reindexing.
>          A "compressionMode" string element can be added to <codecFactory> to choose
>          between the existing compression modes in the default codec: "BEST_SPEED" (default)
>          or "BEST_COMPRESSION".
>     -->
>     <codecFactory class="solr.SchemaCodecFactory"/>
>
>     <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>          Index Config - These settings control low-level behavior of indexing
>          Most example settings here show the default value, but are commented
>          out, to more easily see where customizations have been made.
>
>          Note: This replaces <indexDefaults> and <mainIndex> from older versions
>          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
>     <indexConfig>
>
>         <!-- LockFactory
>
>              This option specifies which Lucene LockFactory implementation
>              to use.
>
>              single = SingleInstanceLockFactory - suggested for a
>                       read-only index or when there is no possibility of
>                       another process trying to modify the index.
>              native = NativeFSLockFactory - uses OS native file locking.
>                       Do not use when multiple solr webapps in the same
>                       JVM are attempting to share a single index.
>              simple = SimpleFSLockFactory  - uses a plain file for locking
>
>              Defaults: 'native' is default for Solr3.6 and later, otherwise
>                        'simple' is the default
>
>              More details on the nuances of each LockFactory...
>              http://wiki.apache.org/lucene-java/AvailableLockFactories
>         -->
>         <lockType>${solr.lock.type:native}</lockType>
>
>         <!-- Lucene Infostream
>
>              To aid in advanced debugging, Lucene provides an "InfoStream"
>              of detailed information when indexing.
>
>              Setting the value to true will instruct the underlying Lucene
>              IndexWriter to write its info stream to solr's log. By default,
>              this is enabled here, and controlled through log4j.properties.
>           -->
>         <infoStream>true</infoStream>
>     </indexConfig>
>
>
>     <!-- JMX
>
>          This example enables JMX if and only if an existing MBeanServer
>          is found, use this if you want to configure JMX through JVM
>          parameters. Remove this to disable exposing Solr configuration
>          and statistics to JMX.
>
>          For more details see http://wiki.apache.org/solr/SolrJmx
>       -->
>     <jmx/>
>     <!-- If you want to connect to a particular server, specify the
>          agentId
>       -->
>     <!-- <jmx agentId="myAgent" /> -->
>     <!-- If you want to start a new MBeanServer, specify the serviceUrl -->
>     <!-- <jmx serviceUrl="service:jmx:rmi:///jndi/rmi://localhost:9999/solr"/>
>       -->
>
>     <!-- The default high-performance update handler -->
>     <updateHandler class="solr.DirectUpdateHandler2">
>
>         <!-- Enables a transaction log, used for real-time get, durability, and
>              and solr cloud replica recovery.  The log can grow as big as
>              uncommitted changes to the index, so use of a hard autoCommit
>              is recommended (see below).
>              "dir" - the target directory for transaction logs, defaults to the
>                     solr data directory.
>              "numVersionBuckets" - sets the number of buckets used to keep
>                     track of max version values when checking for re-ordered
>                     updates; increase this value to reduce the cost of
>                     synchronizing access to version buckets during high-volume
>                     indexing, this requires 8 bytes (long) * numVersionBuckets
>                     of heap space per Solr core.
>         -->
>         <updateLog>
>             <str name="dir">${solr.ulog.dir:}</str>
>             <int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
>         </updateLog>
>
>         <!-- AutoCommit
>
>              Perform a hard commit automatically under certain conditions.
>              Instead of enabling autoCommit, consider using "commitWithin"
>              when adding documents.
>
>              http://wiki.apache.org/solr/UpdateXmlMessages
>
>              maxDocs - Maximum number of documents to add since the last
>                        commit before automatically triggering a new commit.
>
>              maxTime - Maximum amount of time in ms that is allowed to pass
>                        since a document was added before automatically
>                        triggering a new commit.
>              openSearcher - if false, the commit causes recent index changes
>                to be flushed to stable storage, but does not cause a new
>                searcher to be opened to make those changes visible.
>
>              If the updateLog is enabled, then it's highly recommended to
>              have some sort of hard autoCommit to limit the log size.
>           -->
>         <autoCommit>
>             <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>             <openSearcher>false</openSearcher>
>         </autoCommit>
>
>         <!-- softAutoCommit is like autoCommit except it causes a
>              'soft' commit which only ensures that changes are visible
>              but does not ensure that data is synced to disk.  This is
>              faster and more near-realtime friendly than a hard commit.
>           -->
>         <!-- setting softAutoCommit to 15 minutes for fft transcribedReports usage.          -->
>         <autoSoftCommit>
>             <maxTime>${solr.autoSoftCommit.maxTime:900000}</maxTime>
>         </autoSoftCommit>
>
>     </updateHandler>
>
>     <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>          Query section - these settings control query time things like caches
>          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
>     <query>
>         <!-- Max Boolean Clauses
>
>              Maximum number of clauses in each BooleanQuery,  an exception
>              is thrown if exceeded.
>
>              ** WARNING **
>
>              This option actually modifies a global Lucene property that
>              will affect all SolrCores.  If multiple solrconfig.xml files
>              disagree on this property, the value at any given moment will
>              be based on the last SolrCore to be initialized.
>
>           -->
>         <maxBooleanClauses>1024</maxBooleanClauses>
>
>
>         <!-- Solr Internal Query Caches
>
>              There are two implementations of cache available for Solr,
>              LRUCache, based on a synchronized LinkedHashMap, and
>              FastLRUCache, based on a ConcurrentHashMap.
>
>              FastLRUCache has faster gets and slower puts in single
>              threaded operation and thus is generally faster than LRUCache
>              when the hit ratio of the cache is high (> 75%), and may be
>              faster under other scenarios on multi-cpu systems.
>         -->
>
>         <!-- Filter Cache
>
>              Cache used by SolrIndexSearcher for filters (DocSets),
>              unordered sets of *all* documents that match a query.  When a
>              new searcher is opened, its caches may be prepopulated or
>              "autowarmed" using data from caches in the old searcher.
>              autowarmCount is the number of items to prepopulate.  For
>              LRUCache, the autowarmed items will be the most recently
>              accessed items.
>
>              Parameters:
>                class - the SolrCache implementation LRUCache or
>                    (LRUCache or FastLRUCache)
>                size - the maximum number of entries in the cache
>                initialSize - the initial capacity (number of entries) of
>                    the cache.  (see java.util.HashMap)
>                autowarmCount - the number of entries to prepopulate from
>                    and old cache.
>           -->
>         <filterCache class="solr.FastLRUCache"
>                      size="512"
>                      initialSize="512"
>                      autowarmCount="0"/>
>
>         <!-- Query Result Cache
>
>             Caches results of searches - ordered lists of document ids
>             (DocList) based on a query, a sort, and the range of documents requested.
>             Additional supported parameter by LRUCache:
>                maxRamMB - the maximum amount of RAM (in MB) that this cache is allowed
>                           to occupy
>          -->
>         <queryResultCache class="solr.LRUCache"
>                           size="512"
>                           initialSize="512"
>                           autowarmCount="0"/>
>
>         <!-- Document Cache
>
>              Caches Lucene Document objects (the stored fields for each
>              document).  Since Lucene internal document ids are transient,
>              this cache will not be autowarmed.
>           -->
>         <documentCache class="solr.LRUCache"
>                        size="512"
>                        initialSize="512"
>                        autowarmCount="0"/>
>
>         <!-- custom cache currently used by block join -->
>         <cache name="perSegFilter"
>                class="solr.search.LRUCache"
>                size="10"
>                initialSize="0"
>                autowarmCount="10"
>                regenerator="solr.NoOpRegenerator"/>
>
>         <!-- Lazy Field Loading
>
>              If true, stored fields that are not requested will be loaded
>              lazily.  This can result in a significant speed improvement
>              if the usual case is to not load all stored fields,
>              especially if the skipped fields are large compressed text
>              fields.
>         -->
>         <enableLazyFieldLoading>true</enableLazyFieldLoading>
>
>         <!-- Result Window Size
>
>              An optimization for use with the queryResultCache.  When a search
>              is requested, a superset of the requested number of document ids
>              are collected.  For example, if a search for a particular query
>              requests matching documents 10 through 19, and queryWindowSize is 50,
>              then documents 0 through 49 will be collected and cached.  Any further
>              requests in that range can be satisfied via the cache.
>           -->
>         <queryResultWindowSize>20</queryResultWindowSize>
>
>         <!-- Maximum number of documents to cache for any entry in the
>              queryResultCache.
>           -->
>         <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
>
>         <!-- Use Cold Searcher
>
>              If a search request comes in and there is no current
>              registered searcher, then immediately register the still
>              warming searcher and use it.  If "false" then all requests
>              will block until the first searcher is done warming.
>           -->
>         <useColdSearcher>false</useColdSearcher>
>
>         <!-- Max Warming Searchers
>
>              Maximum number of searchers that may be warming in the
>              background concurrently.  An error is returned if this limit
>              is exceeded.
>
>              Recommend values of 1-2 for read-only slaves, higher for
>              masters w/o cache warming.
>           -->
>         <maxWarmingSearchers>2</maxWarmingSearchers>
>
>     </query>
>
>
>     <!-- Request Dispatcher
>
>          This section contains instructions for how the SolrDispatchFilter
>          should behave when processing requests for this SolrCore.
>
>          handleSelect is a legacy option that affects the behavior of requests
>          such as /select?qt=XXX
>
>          handleSelect="true" will cause the SolrDispatchFilter to process
>          the request and dispatch the query to a handler specified by the
>          "qt" param, assuming "/select" isn't already registered.
>
>          handleSelect="false" will cause the SolrDispatchFilter to
>          ignore "/select" requests, resulting in a 404 unless a handler
>          is explicitly registered with the name "/select"
>
>          handleSelect="true" is not recommended for new users, but is the default
>          for backwards compatibility
>       -->
>     <requestDispatcher handleSelect="false">
>         <!-- Request Parsing
>
>              These settings indicate how Solr Requests may be parsed, and
>              what restrictions may be placed on the ContentStreams from
>              those requests
>
>              enableRemoteStreaming - enables use of the stream.file
>              and stream.url parameters for specifying remote streams.
>
>              multipartUploadLimitInKB - specifies the max size (in KiB) of
>              Multipart File Uploads that Solr will allow in a Request.
>
>              formdataUploadLimitInKB - specifies the max size (in KiB) of
>              form data (application/x-www-form-urlencoded) sent via
>              POST. You can use POST to pass request parameters not
>              fitting into the URL.
>
>              addHttpRequestToContext - if set to true, it will instruct
>              the requestParsers to include the original HttpServletRequest
>              object in the context map of the SolrQueryRequest under the
>              key "httpRequest". It will not be used by any of the existing
>              Solr components, but may be useful when developing custom
>              plugins.
>
>              *** WARNING ***
>              The settings below authorize Solr to fetch remote files, You
>              should make sure your system has some authentication before
>              using enableRemoteStreaming="true"
>
>           -->
>         <requestParsers enableRemoteStreaming="true"
>                         multipartUploadLimitInKB="2048000"
>                         formdataUploadLimitInKB="2048"
>                         addHttpRequestToContext="false"/>
>
>         <!-- HTTP Caching
>
>              Set HTTP caching related parameters (for proxy caches and clients).
>
>              The options below instruct Solr not to output any HTTP Caching
>              related headers
>           -->
>         <httpCaching never304="true"/>
>
>     </requestDispatcher>
>
>     <!-- Request Handlers
>
>          http://wiki.apache.org/solr/SolrRequestHandler
>
>          Incoming queries will be dispatched to a specific handler by name
>          based on the path specified in the request.
>
>          Legacy behavior: If the request path uses "/select" but no Request
>          Handler has that name, and if handleSelect="true" has been specified in
>          the requestDispatcher, then the Request Handler is dispatched based on
>          the qt parameter.  Handlers without a leading '/' are accessed this way
>          like so: http://host/app/[core/]select?qt=name  If no qt is
>          given, then the requestHandler that declares default="true" will be
>          used or the one named "standard".
>
>          If a Request Handler is declared with startup="lazy", then it will
>          not be initialized until the first request that uses it.
>
>       -->
>     <!-- SearchHandler
>
>          http://wiki.apache.org/solr/SearchHandler
>
>          For processing Search Queries, the primary Request Handler
>          provided with Solr is "SearchHandler" It delegates to a sequent
>          of SearchComponents (see below) and supports distributed
>          queries across multiple shards
>       -->
>     <requestHandler name="/select" class="solr.SearchHandler">
>         <!-- default values for query parameters can be specified, these
>              will be overridden by parameters in the request
>           -->
>         <lst name="defaults">
>             <str name="echoParams">explicit</str>
>             <int name="rows">10</int>
>         </lst>
>
>     </requestHandler>
>
>     <!-- A request handler that returns indented JSON by default -->
>     <requestHandler name="/query" class="solr.SearchHandler">
>         <lst name="defaults">
>             <str name="echoParams">explicit</str>
>             <str name="wt">json</str>
>             <str name="indent">true</str>
>             <str name="df">text</str>
>         </lst>
>     </requestHandler>
>
>     <initParams path="/update/**,/query,/select,/tvrh,/elevate,/spell">
>         <lst name="defaults">
>             <str name="df">text</str>
>         </lst>
>     </initParams>
>
>     <!-- Field Analysis Request Handler
>
>          RequestHandler that provides much the same functionality as
>          analysis.jsp. Provides the ability to specify multiple field
>          types and field names in the same request and outputs
>          index-time and query-time analysis for each of them.
>
>          Request parameters are:
>          analysis.fieldname - field name whose analyzers are to be used
>
>          analysis.fieldtype - field type whose analyzers are to be used
>          analysis.fieldvalue - text for index-time analysis
>          q (or analysis.q) - text for query time analysis
>          analysis.showmatch (true|false) - When set to true and when
>              query analysis is performed, the produced tokens of the
>              field value analysis will be marked as "matched" for every
>              token that is produces by the query analysis
>      -->
>     <requestHandler name="/analysis/field"
>                     startup="lazy"
>                     class="solr.FieldAnalysisRequestHandler"/>
>
>
>     <!-- Document Analysis Handler
>
>          http://wiki.apache.org/solr/AnalysisRequestHandler
>
>          An analysis handler that provides a breakdown of the analysis
>          process of provided documents. This handler expects a (single)
>          content stream with the following format:
>
>          <docs>
>            <doc>
>              <field name="id">1</field>
>              <field name="name">The Name</field>
>              <field name="text">The Text Value</field>
>            </doc>
>            <doc>...</doc>
>            <doc>...</doc>
>            ...
>          </docs>
>
>       Note: Each document must contain a field which serves as the
>       unique key. This key is used in the returned response to associate
>       an analysis breakdown to the analyzed document.
>
>       Like the FieldAnalysisRequestHandler, this handler also supports
>       query analysis by sending either an "analysis.query" or "q"
>       request parameter that holds the query text to be analyzed. It
>       also supports the "analysis.showmatch" parameter which when set to
>       true, all field tokens that match the query tokens will be marked
>       as a "match".
>     -->
>     <requestHandler name="/analysis/document"
>                     class="solr.DocumentAnalysisRequestHandler"
>                     startup="lazy"/>
>
>     <!-- Echo the request contents back to the client -->
>     <requestHandler name="/debug/dump" class="solr.DumpRequestHandler">
>         <lst name="defaults">
>             <str name="echoParams">explicit</str>
>             <str name="echoHandler">true</str>
>         </lst>
>     </requestHandler>
>
>
>     <!-- Search Components
>
>          Search components are registered to SolrCore and used by
>          instances of SearchHandler (which can access them by name)
>
>          By default, the following components are available:
>
>          <searchComponent name="query"     class="solr.QueryComponent" />
>          <searchComponent name="facet"     class="solr.FacetComponent" />
>          <searchComponent name="mlt"       class="solr.MoreLikeThisComponent" />
>          <searchComponent name="highlight" class="solr.HighlightComponent" />
>          <searchComponent name="stats"     class="solr.StatsComponent" />
>          <searchComponent name="debug"     class="solr.DebugComponent" />
>
>        -->
>     <!-- Update Processors
>
>        Chains of Update Processor Factories for dealing with Update
>        Requests can be declared, and then used by name in Update
>        Request Processors
>
>        http://wiki.apache.org/solr/UpdateRequestProcessor
>
>     -->
>
>     <!-- This processing chain handles document expiry.
>         The fft task that exports to solr specifies an expiryDate field
>         and this configures solr to run a thread to delete
>         documents that have passed their expiryDate
>     -->
>     <updateRequestProcessorChain name="FFTtranscribedReportsProcessorChain" default="true">
>         <processor class="solr.IgnoreCommitOptimizeUpdateProcessorFactory">
>           <int name="statusCode">200</int>
>         </processor>
>         <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
>             <int name="autoDeletePeriodSeconds">30</int>
>             <str name="ttlFieldName">_ttl_</str>
>             <str name="expirationFieldName">expiryDate</str>
>         </processor>
>         <processor class="solr.FirstFieldValueUpdateProcessorFactory">
>             <str name="fieldName">expiryDate</str>
>         </processor>
>         <processor class="solr.LogUpdateProcessorFactory" />
>         <processor class="solr.DistributedUpdateProcessorFactory" />
>         <processor class="solr.RunUpdateProcessorFactory" />
>     </updateRequestProcessorChain>
>
>      <!-- Terms Component
>
>          http://wiki.apache.org/solr/TermsComponent
>
>          A component to return terms and document frequency of those
>          terms
>     -->
>     <searchComponent name="terms" class="solr.TermsComponent"/>
>
>     <!-- A request handler for demonstrating the terms component -->
>     <requestHandler name="/terms" class="solr.SearchHandler" startup="lazy">
>         <lst name="defaults">
>             <bool name="terms">true</bool>
>             <bool name="distrib">false</bool>
>         </lst>
>         <arr name="components">
>             <str>terms</str>
>         </arr>
>     </requestHandler>
>
> </config>
>
>
>
>
>
>
>
> -----Original Message-----
> From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
> Sent: Friday, September 2, 2016 5:36 AM
> To: solr-user@lucene.apache.org
> Subject: Re: what to try next for replica that will not stay up.
>
> My guess is that recovery is indeed successful but the leader is repeatedly marking this replicas as 'down' using what we call Leader-Initiated-Recovery or LIR. We need to understand why that is happening. Are there GC issues on this new node?
>
> Can we see the logs on the leader from a bit before and after a line that should look like the following:
> "Put replica core=documents_shard1_replica2 coreNodeName="...
>
> Can we also see the complete cluster state?
>
> On Thu, Sep 1, 2016 at 9:15 PM, Jon Hawkesworth < jon.hawkesworth@medquist.onmicrosoft.com> wrote:
>
>> Hi
>>
>>
>>
>> If anyone has any suggestions of things I could try to resolve my
>> issue where one replica on one of my solcloud 6.0.1 shards refuses to
>> stay up, I'd love to hear them.  In fact, I'll get you something off
>> your amazon wishlist, within reason, if you can solve this puzzle.
>>
>>
>>
>> Today we pruned the dead replica, restarted the machine where it ran
>> and once the node had rejoined the cluster, we added a new replica.
>>
>> The replica was marked as Active for about 10 minutes then went down
>>
>>
>>
>> I put some example logging from below, but it looks much the same as
>> last time.
>>
>>
>>
>> There's a bunch of warnings about a checksum being different even
>> though the file size is the same and then RecoveryStrategy
>>
>> reports 'Could not publish as ACTIVE after succesful recovery'
>>
>>
>>
>> I think I've found where that message comes from in the code here:
>> https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=
>> blob;f=solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java;
>> h= abd00aef19a731b42b314f8b526cdb2d77baf89f;hb=refs/heads/master
>>
>> (I am running 6.0.1 though so could have changed in latest devel).
>>
>>
>>
>> So it seems this chunk of codeā€¦
>>
>>
>>
>> 451         if (successfulRecovery) {
>>
>> 452           LOG.info("Registering as Active after recovery.");
>>
>> 453           try {
>>
>> 454             zkController.publish(core.getCoreDescriptor(),
>> Replica.State.ACTIVE);
>>
>> 455           } catch (Exception e) {
>>
>> 456             LOG.error("Could not publish as ACTIVE after succesful
>> recovery", e);
>>
>> 457             successfulRecovery = false;
>>
>> 458           }
>>
>> 459
>>
>>  460           if (successfulRecovery) {
>>
>> 461             close = true;
>>
>> 462             recoveryListener.recovered();
>>
>> 463           }
>>
>> 464         }
>>
>>
>>
>> results in this:
>>
>>
>>
>> org.apache.solr.common.SolrException: Cannot publish state of core
>> 'documents_shard1_replica2' as active without recovering first!
>>
>>                at org.apache.solr.cloud.ZkController.publish(
>> ZkController.java:1141)
>>
>>                at org.apache.solr.cloud.ZkController.publish(
>> ZkController.java:1097)
>>
>>                at org.apache.solr.cloud.ZkController.publish(
>> ZkController.java:1093)
>>
>>                at org.apache.solr.cloud.RecoveryStrategy.doRecovery(
>> RecoveryStrategy.java:457)
>>
>>                at org.apache.solr.cloud.RecoveryStrategy.run(
>> RecoveryStrategy.java:224)
>>
>>                at
>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown
>> Source)
>>
>>                at java.util.concurrent.FutureTask.run(Unknown Source)
>>
>>                at org.apache.solr.common.util.ExecutorUtil$
>> MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>>
>>                at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>> Source)
>>
>>                at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
>> Source)
>>
>>                at java.lang.Thread.run(Unknown Source)
>>
>>
>>
>> I don't yet understand the interaction with zookeeper but there's some
>> disagreement about whether recovery has happened or not (if it hadn't
>> from solr's point of view the successfulRecovery boolean would
>> presumably be false.
>>
>>
>>
>> Should I raise a JIRA?  Is there any other useful information I could
>> gather?
>>
>>
>>
>> I haven't really had any similar problems with the other 3 shards,
>> just shard1.
>>
>>
>>
>> The nodes that it is running on are all pretty similar - all vms built
>> to the same specification and the deployment of java and solrcloud is
>> automated so there shouldn't be any differences in the stack.
>>
>>
>>
>> Many thanks,
>>
>>
>>
>> Jon
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Example log output below
>>
>>
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnux.si did not match. expected checksum is 1186898951 and
>> actual is checksum 1994281621. expected length is 417 and actual
>> length is 417
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy.nvd did not match. expected checksum is 2200422612 and
>> actual is checksum 3635321041. expected length is 63 and actual length
>> is 65
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy.fdx did not match. expected checksum is 281622189 and
>> actual is checksum 838341528. expected length is 84 and actual length
>> is 84
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy.nvm did not match. expected checksum is 1875012021 and
>> actual is checksum 524812847. expected length is 108 and actual length
>> is 108
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy.fnm did not match. expected checksum is 1681449973 and
>> actual is checksum 3351426142. expected length is 1265 and actual
>> length is 1265
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy_Lucene54_0.dvm did not match. expected checksum is
>> 355987228 and actual is checksum 847034886. expected length is 380 and
>> actual length is 404
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy_Lucene50_0.pos did not match. expected checksum is
>> 806636274 and actual is checksum 2272195325. expected length is 1059
>> and actual length is 1172
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy_Lucene50_0.doc did not match. expected checksum is
>> 4041316671 and actual is checksum 3122885740. expected length is 212
>> and actual length is 281
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy_Lucene50_0.tim did not match. expected checksum is
>> 2891628412 and actual is checksum 2420913910. expected length is 5346
>> and actual length is 6251
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy_Lucene50_0.tip did not match. expected checksum is
>> 1652105503 and actual is checksum 807238796. expected length is 336
>> and actual length is 349
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy_Lucene54_0.dvd did not match. expected checksum is
>> 2664049801 and actual is checksum 2930561414. expected length is 130
>> and actual length is 145
>>
>> 9/1/2016, 12:37:06 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy.fdt did not match. expected checksum is 4175958592 and
>> actual is checksum 3650490510. expected length is 4280 and actual
>> length is 4983
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuy.si did not match. expected checksum is 2223401636 and
>> actual is checksum 734463570. expected length is 535 and actual length
>> is 535
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz_Lucene54_0.dvd did not match. expected checksum is
>> 202072236 and actual is checksum 4194802930. expected length is 96 and
>> actual length is 264
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz_Lucene50_0.tip did not match. expected checksum is
>> 2123658306 and actual is checksum 435878007. expected length is 298
>> and actual length is 639
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz.nvd did not match. expected checksum is 4214748910 and
>> actual is checksum 3784036105. expected length is 59 and actual length
>> is 77
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz.fdt did not match. expected checksum is 3837568601 and
>> actual is checksum 2542454689. expected length is 896 and actual
>> length is 20338
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz.fdx did not match. expected checksum is 2070429440 and
>> actual is checksum 3279752998. expected length is 84 and actual length
>> is 86
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz_Lucene50_0.pos did not match. expected checksum is
>> 2299588010 and actual is checksum 2299553846. expected length is 190
>> and actual length is 5717
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz_Lucene54_0.dvm did not match. expected checksum is
>> 914650440 and actual is checksum 2852383192. expected length is 312
>> and actual length is 548
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz.nvm did not match. expected checksum is 3037735995 and
>> actual is checksum 4023026424. expected length is 108 and actual
>> length is 108
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz_Lucene50_0.doc did not match. expected checksum is
>> 3813274592 and actual is checksum 189237707. expected length is 110
>> and actual length is 1945
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz_Lucene50_0.tim did not match. expected checksum is
>> 3013245878 and actual is checksum 2122722316. expected length is 1757
>> and actual length is 16642
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz.fnm did not match. expected checksum is 2117653105 and
>> actual is checksum 3401755804. expected length is 1265 and actual
>> length is 1265
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnuz.si did not match. expected checksum is 2715978927 and
>> actual is checksum 3653125964. expected length is 535 and actual
>> length is 535
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0.fdt did not match. expected checksum is 1699019853 and
>> actual is checksum 3731775500. expected length is 15865 and actual
>> length is 12728
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0_Lucene50_0.pos did not match. expected checksum is
>> 2189908204 and actual is checksum 2338139479. expected length is 4475
>> and actual length is 3431
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0_Lucene50_0.doc did not match. expected checksum is
>> 1522019614 and actual is checksum 969681917. expected length is 1394
>> and actual length is 1093
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0_Lucene50_0.tim did not match. expected checksum is
>> 813529901 and actual is checksum 529669468. expected length is 13843
>> and actual length is 12535
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0.si did not match. expected checksum is 3802482417 and
>> actual is checksum 1865633126. expected length is 535 and actual
>> length is 535
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0_Lucene54_0.dvm did not match. expected checksum is
>> 4236057860 and actual is checksum 2986112802. expected length is 500
>> and actual length is 476
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0.fdx did not match. expected checksum is 2497099401 and
>> actual is checksum 990046808. expected length is 85 and actual length
>> is 84
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0.nvd did not match. expected checksum is 1736308969 and
>> actual is checksum 3657480551. expected length is 73 and actual length
>> is 71
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0_Lucene50_0.tip did not match. expected checksum is
>> 1362235492 and actual is checksum 640196019. expected length is 570
>> and actual length is 531
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0.fnm did not match. expected checksum is 1975043794 and
>> actual is checksum 1035049893. expected length is 1265 and actual
>> length is 1265
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0.nvm did not match. expected checksum is 2985228383 and
>> actual is checksum 2603407196. expected length is 108 and actual
>> length is 108
>>
>> 9/1/2016, 12:37:07 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> IndexFetcher
>>
>>
>>
>> File _jnv0_Lucene54_0.dvd did not match. expected checksum is
>> 762056409 and actual is checksum 1514176651. expected length is 228
>> and actual length is 211
>>
>> 9/1/2016, 12:37:09 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> UpdateLog
>>
>>
>>
>> Starting log replay tlog{file=E:\solr_home\transcribedReports_shard1_
>> replica2\data\tlog\tlog.0000000000000000005 refcount=2} active=true
>> starting pos=20222
>>
>> 9/1/2016, 12:37:10 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> UpdateLog
>>
>>
>>
>> Log replay finished. recoveryInfo=RecoveryInfo{adds=10 deletes=0
>> deleteByQuery=1 errors=0 positionOfStart=20222}
>>
>> 9/1/2016, 12:37:10 PM
>>
>>
>>
>> ERROR false
>>
>>
>>
>>
>>
>> RecoveryStrategy
>>
>>
>>
>> Could not publish as ACTIVE after succesful recovery
>>
>> 9/1/2016, 12:37:10 PM
>>
>>
>>
>> ERROR false
>>
>>
>>
>>
>>
>> RecoveryStrategy
>>
>>
>>
>> Recovery failed - trying again... (0)
>>
>> 9/1/2016, 12:37:37 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> UpdateLog
>>
>>
>>
>> Starting log replay tlog{file=E:\solr_home\transcribedReports_shard1_
>> replica2\data\tlog\tlog.0000000000000000006 refcount=2} active=true
>> starting pos=0
>>
>> 9/1/2016, 12:37:38 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> UpdateLog
>>
>>
>>
>> Log replay finished. recoveryInfo=RecoveryInfo{adds=2 deletes=0
>> deleteByQuery=0 errors=0 positionOfStart=0}
>>
>> 9/1/2016, 12:37:41 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> RecoveryStrategy
>>
>>
>>
>> Stopping recovery for core=[transcribedReports_shard1_replica2]
>> coreNodeName=[core_node14]
>>
>> 9/1/2016, 12:42:13 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> UpdateLog
>>
>>
>>
>> Starting log replay tlog{file=E:\solr_home\transcribedReports_shard1_
>> replica2\data\tlog\tlog.0000000000000000007 refcount=2} active=true
>> starting pos=1748
>>
>> 9/1/2016, 12:42:14 PM
>>
>>
>>
>> WARN false
>>
>>
>>
>>
>>
>> UpdateLog
>>
>>
>>
>> Log replay finished. recoveryInfo=RecoveryInfo{adds=12 deletes=0
>> deleteByQuery=0 errors=0 positionOfStart=1748}
>>
>> 9/1/2016, 12:42:14 PM
>>
>>
>>
>> ERROR false
>>
>>
>>
>>
>>
>> RecoveryStrategy
>>
>>
>>
>> Could not publish as ACTIVE after succesful recovery
>>
>> 9/1/2016, 12:42:14 PM
>>
>>
>>
>> ERROR false
>>
>>
>>
>>
>>
>> RecoveryStrategy
>>
>>
>>
>> Recovery failed - trying again... (0)
>>
>> 9/1/2016, 12:42:43 PM
>>
>>
>>
>> ERROR false
>>
>>
>>
>>
>>
>> RecoveryStrategy
>>
>>
>>
>> Could not publish as ACTIVE after succesful recovery
>>
>> 9/1/2016, 12:42:43 PM
>>
>>
>>
>> ERROR false
>>
>>
>>
>>
>>
>> RecoveryStrategy
>>
>>
>>
>> Recovery failed - trying again... (0)
>>
>>
>>
>> *Jon Hawkesworth*
>> Software Developer
>>
>>
>>
>>
>>
>> Hanley Road, Malvern, WR13 6NP. UK
>>
>> O: +44 (0) 1684 312313
>>
>> *jon.hawkesworth@mmodal.com <jon.hawkesworth@mmodal.com>
>> www.mmodal.com
>> <http://www.medquist.com/>*
>>
>>
>>
>> *This electronic mail transmission contains confidential information
>> intended only for the person(s) named. Any use, distribution, copying
>> or disclosure by another person is strictly prohibited. If you are not
>> the intended recipient of this e-mail, promptly delete it and all
>> attachments.*
>>
>>
>>
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.



-- 
-----------------------------------------------------
Noble Paul


Mime
View raw message