lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Hawkesworth <jon.hawkeswo...@MEDQUIST.onmicrosoft.com>
Subject RE: what to try next for replica that will not stay up.
Date Fri, 02 Sep 2016 16:26:26 GMT
A little more info:

Once we start indexing we start seeing a lot of these:

2016-09-02 12:48:29,794 level=INFO    thread=qtp762152757-17 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50 
2016-09-02 12:48:29,794 level=INFO    thread=qtp762152757-17 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2 
2016-09-02 12:48:29,794 level=INFO    thread=qtp762152757-17 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
java.io.IOException: Committed before 100 Continues
	at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
	at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Unknown Source)

We clearly have a *lot* of leader_initiated_recovery happening on shard1 - looking at the 

cversion attribute the /collections/transcribedReports/leader_initiated_recovery/shardX nodes I can see the versions are as follows

shard1 312
shard2 18
shard3 24
shard4 24

Jon


-----Original Message-----
From: Jon Hawkesworth [mailto:jon.hawkesworth@MEDQUIST.onmicrosoft.com] 
Sent: Friday, September 2, 2016 4:14 PM
To: solr-user@lucene.apache.org
Subject: RE: what to try next for replica that will not stay up.

Many thanks for continuing to investigate this, Shalin.  

So we were able to bring the replica up for approx 30 minutes after removing the leader-initiated-recovery/shard1 node from zookeeper and restarting all 4 nodes.  However soon after we enabled indexing the replica quickly went into Down state again.

I am not sure about garbage collection on the node.  It hasn't looked busy cpu wise and java threads usually seem to be around 50 when I check resource monitor, but I have included the logging things you asked for below 1/ a few seconds of logging from either side of the 'Put replica core=' (the collection is actually called transcribedReports) shortly after we re-enabled indexing and the replica went into Down state, 2/ clusterstate.json, 3/ the collection's solrconfig.xml.  I think the only changes to default solrconfig.xml are for the updateRequestProcessorChain (ignore commits from clients and auto delete expired documents - is every 30seconds too frequent for this?), and auto commit configuration (hard autocommit 15 seconds, opensearch=false soft autocommit every 15 *minutes*).

I note some 401 Unauthorised request, which is odd since security.xml is applied to the cluster.  Also some 'java.io.IOException: Committed before 100 Continues' exceptions which may or may not be of interest.  I am speculating that the 401s between nodes is stopping updates based on the contents of the transaction log, causing it to fall back to old full replication.  While there are no incoming documents to process it is able to keep up with old full replication, but its too much for it to handle when there are new documents arriving all the time (I estimate the cluster recieves 1-2 documents per second on average during the day).

I can send full log from startup, adding the replica and then turning on ingestion if that would be helpful (3+Mb though, not suitable for a mailing list I think).  Please let me know if you need any further information.

Jon



1/ (logging)

2016-09-02 12:49:06,138 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol003:8983/solr/transcribedReports_shard3_replica3/&wt=javabin&version=2}{add=[711930167 (1544364276970422272)]} 0 22
2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
2016-09-02 12:49:07,310 level=INFO    thread=qtp762152757-13 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=6
2016-09-02 12:49:10,372 level=INFO    thread=qtp762152757-21 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
2016-09-02 12:49:10,388 level=INFO    thread=qtp762152757-21 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
2016-09-02 12:49:10,388 level=INFO    thread=qtp762152757-21 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
2016-09-02 12:49:10,388 level=INFO    thread=qtp762152757-21 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
2016-09-02 12:49:10,388 level=INFO    thread=qtp762152757-21 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=6
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.DocExpirationUpdateProcessorFactory Begining periodic deletion of expired docs
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: anyChanges? numDocsInRam=5 deletes=true hasTickets:false pendingChangesInFullFlush: false
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: nrtIsCurrent: infoVersion matches: false; DW changes: true; BD changes: false
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: flush at getReader
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: startFullFlush
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: anyChanges? numDocsInRam=5 deletes=true hasTickets:false pendingChangesInFullFlush: false
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWFC][autoExpireDocs-18-thread-1]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 4 deleted terms (unique count=4) bytesUsed=692, segment=_jtqb, aborted=false, numDocsInRAM=5, deleteQueue=DWDQ: [ generation: 91 ]]
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flush postings as segment _jtqb numDocs=5
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 1 msec to write norms
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 1 msec to write docValues
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 0 msec to write points
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 1 msec to finish stored fields
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 6 msec to write postings and finish vectors
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: 0 msec to write fieldInfos
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: new segment has 0 deleted docs
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: new segment has no vectors; norms; docValues; prox; freqs
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flushedFiles=[_jtqb.fdt, _jtqb.nvm, _jtqb.fnm, _jtqb_Lucene50_0.pos, _jtqb.fdx, _jtqb_Lucene54_0.dvm, _jtqb_Lucene50_0.doc, _jtqb_Lucene50_0.tim, _jtqb_Lucene54_0.dvd, _jtqb_Lucene50_0.tip, _jtqb.nvd]
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flushed codec=Lucene60
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flushed: segment=_jtqb ramUsed=0.269 MB newFlushedSize=0.021 MB docs/MB=243.379
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][autoExpireDocs-18-thread-1]: flush time 11.28663 msec
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: publishFlushedSegment seg-private updates= 4 deleted terms (unique count=0) 1 deleted queries bytesUsed=80
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: flush: push buffered seg private updates:  4 deleted terms (unique count=0) 1 deleted queries bytesUsed=80
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: publishFlushedSegment
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: push deletes  5 deleted terms (unique count=5) 1 deleted queries bytesUsed=1104 segmentPrivate?=false delGen=181 packetCount=1 totBytesUsed=1104
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: push deletes  4 deleted terms (unique count=0) 1 deleted queries bytesUsed=80 segmentPrivate?=true delGen=182 packetCount=2 totBytesUsed=1184
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: publish sets newSegment delGen=182 seg=_jtqb(6.0.1):C5
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][autoExpireDocs-18-thread-1]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5" [25 segments ; isCommit = false]
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][autoExpireDocs-18-thread-1]: delete []
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][autoExpireDocs-18-thread-1]: 0 msec to checkpoint
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: apply all deletes during flush
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: now apply all deletes for all segments maxDoc=1450423
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyDeletes: open segment readers took 0 msec
2016-09-02 12:49:11,794 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyDeletes: infos=[_5tw9(6.0.1):C241588/706:delGen=703, _fb82(6.0.1):C387586/788:delGen=727, _iin7(6.0.1):C491496/573:delGen=526, _i0at(6.0.1):c17859/119:delGen=112, _iy15(6.0.1):c32975/162:delGen=157, _iqm7(6.0.1):c67369/342:delGen=325, _jf5b(6.0.1):c22457/84:delGen=83, _j7ia(6.0.1):c65875/307:delGen=272, _jksn(6.0.1):c60239/75:delGen=70, _jthf(6.0.1):c27514/3:delGen=3, _jmdv(6.0.1):c7804/90:delGen=80, _jo5r(6.0.1):c7330/119:delGen=93, _jp6z(6.0.1):c7377/43:delGen=36, _jq41(6.0.1):c6791/32:delGen=31, _jquf(6.0.1):c5259/7:delGen=7, _jtpr(6.0.1):c119, _jtox(6.0.1):c108, _jtod(6.0.1):c97, _jtph(6.0.1):c114, _jtq1(6.0.1):c117, _jto3(6.0.1):c98, _jtp7(6.0.1):c123, _jton(6.0.1):c118, _jtqa(6.0.1):C5, _jtqb(6.0.1):C5] packetCount=2
2016-09-02 12:49:11,810 level=WARN    thread=updateExecutor-2-thread-1-processing-http:////atlprwfftsol003:8983//solr//transcribedReports_shard3_replica3 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient Failed to parse error response from http://atlprwfftsol003:8983/solr/transcribedReports_shard3_replica3 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
2016-09-02 12:49:11,810 level=ERROR   thread=updateExecutor-2-thread-1-processing-http:////atlprwfftsol003:8983//solr//transcribedReports_shard3_replica3 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.update.StreamingSolrClients error
org.apache.solr.common.SolrException: Unauthorized request, Response code: 401



request: http://atlprwfftsol003:8983/solr/transcribedReports_shard3_replica3/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:11,810 level=WARN    thread=updateExecutor-2-thread-8-processing-http:////atlprwfftsol003:8983//solr//transcribedReports_shard4_replica4 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient Failed to parse error response from http://atlprwfftsol003:8983/solr/transcribedReports_shard4_replica4 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
2016-09-02 12:49:11,810 level=ERROR   thread=updateExecutor-2-thread-8-processing-http:////atlprwfftsol003:8983//solr//transcribedReports_shard4_replica4 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.update.StreamingSolrClients error
org.apache.solr.common.SolrException: Unauthorized request, Response code: 401



request: http://atlprwfftsol003:8983/solr/transcribedReports_shard4_replica4/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyTermDeletes took 1.7 msec for 25 segments and 1 packets; 5 del terms visited; 67 seg terms visited
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyDeletes: 0 new deleted documents
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: applyDeletes took 62 msec for 25 segments, 0 newly deleted docs (query deletes), 5 visited terms, allDeleted=null
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: prune sis=segments_htjo: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 minGen=183 packetCount=2
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][autoExpireDocs-18-thread-1]: pruneDeletes: prune 2 packets; 0 packets remain
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: incRefDeleter for NRT reader version=3536052 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: return reader version=3536052 reader=StandardDirectoryReader(segments_htjo:3536052:nrt _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5)
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][autoExpireDocs-18-thread-1]: autoExpireDocs-18-thread-1 finishFullFlush success=true
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]: findMerges: 25 segments
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_iin7(6.0.1):C491496/573:delGen=526 size=1181.402 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_fb82(6.0.1):C387586/788:delGen=727 size=933.087 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_5tw9(6.0.1):C241588/706:delGen=703 size=580.202 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_j7ia(6.0.1):c65875/307:delGen=272 size=162.085 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_iqm7(6.0.1):c67369/342:delGen=325 size=161.505 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jksn(6.0.1):c60239/75:delGen=70 size=144.827 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_iy15(6.0.1):c32975/162:delGen=157 size=78.643 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jthf(6.0.1):c27514/3:delGen=3 size=66.931 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jf5b(6.0.1):c22457/84:delGen=83 size=54.829 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_i0at(6.0.1):c17859/119:delGen=112 size=42.784 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jmdv(6.0.1):c7804/90:delGen=80 size=20.166 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jo5r(6.0.1):c7330/119:delGen=93 size=18.428 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jp6z(6.0.1):c7377/43:delGen=36 size=16.519 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jq41(6.0.1):c6791/32:delGen=31 size=15.395 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jquf(6.0.1):c5259/7:delGen=7 size=12.601 MB
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jton(6.0.1):c118 size=0.389 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtq1(6.0.1):c117 size=0.365 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtp7(6.0.1):c123 size=0.362 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtpr(6.0.1):c119 size=0.358 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtox(6.0.1):c108 size=0.335 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtph(6.0.1):c114 size=0.320 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jto3(6.0.1):c98 size=0.311 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtod(6.0.1):c97 size=0.287 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtqb(6.0.1):C5 size=0.021 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   seg=_jtqa(6.0.1):C5 size=0.019 MB [floored]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][autoExpireDocs-18-thread-1]:   allowedSegmentCount=31 vs count=25 (eligible count=25) tooBigCount=0
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][autoExpireDocs-18-thread-1]: now merge
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][autoExpireDocs-18-thread-1]:   index: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][autoExpireDocs-18-thread-1]:   no more merges pending; now return
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: getReader took 62 msec
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@7b51b06[transcribedReports_shard1_replica1] realtime]
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][autoExpireDocs-18-thread-1]: delete []
2016-09-02 12:49:11,856 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][autoExpireDocs-18-thread-1]: decRefDeleter for NRT reader version=3536050 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5
2016-09-02 12:49:11,856 level=WARN    thread=updateExecutor-2-thread-7-processing-http:////atlprwfftsol002:8983//solr//transcribedReports_shard2_replica2 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient Failed to parse error response from http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
2016-09-02 12:49:11,856 level=ERROR   thread=updateExecutor-2-thread-7-processing-http:////atlprwfftsol002:8983//solr//transcribedReports_shard2_replica2 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.update.StreamingSolrClients error
org.apache.solr.common.SolrException: Unauthorized request, Response code: 401



request: http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:11,872 level=WARN    thread=updateExecutor-2-thread-1-processing-http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient Failed to parse error response from http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2 due to: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
2016-09-02 12:49:11,872 level=ERROR   thread=updateExecutor-2-thread-1-processing-http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2 x:transcribedReports_shard1_replica1 r:core_node13 n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.update.StreamingSolrClients error
org.apache.solr.common.SolrException: Unauthorized request, Response code: 401



request: http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/update?_version_=-1544364282917945344&update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:11,872 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
2016-09-02 12:49:11,872 level=ERROR   thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.DistributedUpdateProcessor Setting up to try to start recovery on replica http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/
org.apache.solr.common.SolrException: Unauthorized request, Response code: 401



request: http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/update?_version_=-1544364282917945344&update.distrib=FROMLEADER&distrib.from=http%3A%2F%2Fatlprwfftsol001%3A8983%2Fsolr%2FtranscribedReports_shard1_replica1%2F&wt=javabin&version=2
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:290)
	at org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:161)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:11,872 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.cloud.ZkController Put replica core=transcribedReports_shard1_replica2 coreNodeName=core_node14 on atlprwfftsol002:8983_solr into leader-initiated recovery.
2016-09-02 12:49:11,872 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1] {deleteByQuery={!cache=false}expiryDate:[* TO 2016-09-02T12:49:11.794Z] (-1544364282917945344)} 0 82
2016-09-02 12:49:11,872 level=INFO    thread=autoExpireDocs-18-thread-1 category=org.apache.solr.update.processor.DocExpirationUpdateProcessorFactory Finished periodic deletion of expired docs
2016-09-02 12:49:11,872 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.common.cloud.SolrZkClient makePath: /collections/transcribedReports/leader_initiated_recovery/shard1
2016-09-02 12:49:11,903 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.ZkController Wrote down to /collections/transcribedReports/leader_initiated_recovery/shard1/core_node14
2016-09-02 12:49:11,903 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread Put replica core=transcribedReports_shard1_replica2 coreNodeName=core_node14 on atlprwfftsol002:8983_solr into leader-initiated recovery.
2016-09-02 12:49:11,903 level=WARN    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread Leader is publishing core=transcribedReports_shard1_replica2 coreNodeName =core_node14 state=down on behalf of un-reachable replica http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/
2016-09-02 12:49:11,935 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread LeaderInitiatedRecoveryThread-transcribedReports_shard1_replica2 started running to send REQUESTRECOVERY command to http://atlprwfftsol002:8983/solr/transcribedReports_shard1_replica2/; will try for a max of 600 secs
2016-09-02 12:49:11,935 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread Asking core=transcribedReports_shard1_replica2 coreNodeName=core_node14 on http://atlprwfftsol002:8983/solr to recover
2016-09-02 12:49:11,935 level=INFO    thread=zkCallback-4-thread-2-processing-n:atlprwfftsol001:8983_solr category=org.apache.solr.common.cloud.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/transcribedReports/state.json] for collection [transcribedReports] has occurred - updating... (live nodes size: [4])
2016-09-02 12:49:11,935 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread Successfully sent REQUESTRECOVERY command to core=transcribedReports_shard1_replica2 coreNodeName=core_node14 on http://atlprwfftsol002:8983/solr
2016-09-02 12:49:11,935 level=INFO    thread=updateExecutor-2-thread-1-processing-x:transcribedReports_shard1_replica1 r:core_node13 http:////atlprwfftsol002:8983//solr//transcribedReports_shard1_replica2// n:atlprwfftsol001:8983_solr s:shard1 c:transcribedReports category=org.apache.solr.cloud.LeaderInitiatedRecoveryThread LeaderInitiatedRecoveryThread-transcribedReports_shard1_replica2 completed successfully after running for 74.0ms
2016-09-02 12:49:11,950 level=INFO    thread=zkCallback-4-thread-2-processing-n:atlprwfftsol001:8983_solr category=org.apache.solr.common.cloud.ZkStateReader Updating data for [transcribedReports] from [178079] to [178080]
2016-09-02 12:49:11,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation Going to wait for coreNodeName: core_node14, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
2016-09-02 12:49:11,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
2016-09-02 12:49:11,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation Will wait a max of 183 seconds to see transcribedReports_shard1_replica1 (shard1 of transcribedReports) have state: recovering
2016-09-02 12:49:11,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation In WaitForState(recovering): collection=transcribedReports, shard=shard1, thisCore=transcribedReports_shard1_replica1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=atlprwfftsol002:8983_solr, coreNodeName=core_node14, onlyIfActiveCheckResult=false, nodeProps: core_node14:{"state":"down","base_url":"http://atlprwfftsol002:8983/solr","core":"transcribedReports_shard1_replica2","node_name":"atlprwfftsol002:8983_solr"}
2016-09-02 12:49:12,091 level=INFO    thread=zkCallback-4-thread-2-processing-n:atlprwfftsol001:8983_solr category=org.apache.solr.common.cloud.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/transcribedReports/state.json] for collection [transcribedReports] has occurred - updating... (live nodes size: [4])
2016-09-02 12:49:12,091 level=INFO    thread=zkCallback-4-thread-2-processing-n:atlprwfftsol001:8983_solr category=org.apache.solr.common.cloud.ZkStateReader Updating data for [transcribedReports] from [178080] to [178081]
2016-09-02 12:49:12,310 level=INFO    thread=qtp762152757-20 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
2016-09-02 12:49:12,310 level=INFO    thread=qtp762152757-20 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
2016-09-02 12:49:12,310 level=INFO    thread=qtp762152757-20 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
2016-09-02 12:49:12,310 level=INFO    thread=qtp762152757-20 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
2016-09-02 12:49:12,325 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=7
2016-09-02 12:49:12,372 level=INFO    thread=qtp762152757-18 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50 
2016-09-02 12:49:12,372 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2 
2016-09-02 12:49:12,372 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
java.io.IOException: Committed before 100 Continues
	at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
	at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:12,419 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/&wt=javabin&version=2}{add=[711929582 (1544364283546042368)]} 0 6
2016-09-02 12:49:12,700 level=INFO    thread=qtp762152757-19 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
2016-09-02 12:49:12,700 level=INFO    thread=qtp762152757-19 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 1
2016-09-02 12:49:12,794 level=INFO    thread=qtp762152757-15 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol001:8983/solr/transcribedReports_shard2_replica1/&wt=javabin&version=2}{add=[711868996 (1544364283917238272)]} 0 55
2016-09-02 12:49:12,794 level=INFO    thread=qtp762152757-13 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711868996]} 0 68
2016-09-02 12:49:12,794 level=INFO    thread=qtp762152757-21 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/&wt=javabin&version=2}{add=[711816331 (1544364283949744128)]} 0 14
2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50 
2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2 
2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
java.io.IOException: Committed before 100 Continues
	at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
	at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
2016-09-02 12:49:12,810 level=INFO    thread=qtp762152757-18 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 1
2016-09-02 12:49:12,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation In WaitForState(recovering): collection=transcribedReports, shard=shard1, thisCore=transcribedReports_shard1_replica1, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=atlprwfftsol002:8983_solr, coreNodeName=core_node14, onlyIfActiveCheckResult=false, nodeProps: core_node14:{"core":"transcribedReports_shard1_replica2","base_url":"http://atlprwfftsol002:8983/solr","node_name":"atlprwfftsol002:8983_solr","state":"recovering"}
2016-09-02 12:49:12,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.handler.admin.CoreAdminOperation Waited coreNodeName: core_node14, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
2016-09-02 12:49:12,981 level=INFO    thread=qtp762152757-12 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=atlprwfftsol002:8983_solr&onlyIfLeaderActive=true&core=transcribedReports_shard1_replica1&coreNodeName=core_node14&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=997
2016-09-02 12:49:13,216 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol004:8983/solr/transcribedReports_shard3_replica4/&wt=javabin&version=2}{add=[711817959 (1544364284392243200)]} 0 19
2016-09-02 12:49:13,497 level=INFO    thread=qtp762152757-13 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50 
2016-09-02 12:49:13,497 level=INFO    thread=qtp762152757-13 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2 
2016-09-02 12:49:13,497 level=INFO    thread=qtp762152757-13 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
java.io.IOException: Committed before 100 Continues
	at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
	at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:13,560 level=INFO    thread=qtp762152757-21 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/&wt=javabin&version=2}{add=[711926509 (1544364284725690368)]} 0 2
2016-09-02 12:49:13,638 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/info/logging params={wt=json&_=1472817956574&since=0} status=0 QTime=0
2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-20 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol001:8983/solr/transcribedReports_shard2_replica1/&wt=javabin&version=2}{add=[711929853 (1544364285015097344)]} 0 18
2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-12 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711929853]} 0 37
2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-17 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50 
2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-17 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2 
2016-09-02 12:49:13,810 level=INFO    thread=qtp762152757-17 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
java.io.IOException: Committed before 100 Continues
	at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
	at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
2016-09-02 12:49:15,388 level=INFO    thread=qtp762152757-16 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=5
2016-09-02 12:49:17,310 level=INFO    thread=qtp762152757-19 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
2016-09-02 12:49:17,325 level=INFO    thread=qtp762152757-19 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
2016-09-02 12:49:17,325 level=INFO    thread=qtp762152757-19 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
2016-09-02 12:49:17,325 level=INFO    thread=qtp762152757-19 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
2016-09-02 12:49:17,325 level=INFO    thread=qtp762152757-19 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=6
2016-09-02 12:49:17,716 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
2016-09-02 12:49:17,716 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 3
2016-09-02 12:49:18,028 level=INFO    thread=qtp762152757-12 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
2016-09-02 12:49:18,028 level=INFO    thread=qtp762152757-12 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 1
2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-21 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol001:8983/solr/transcribedReports_shard2_replica1/&wt=javabin&version=2}{add=[711834143 (1544364289454768128)]} 0 19
2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-18 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711834143]} 0 41
2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-20 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50 
2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2 
2016-09-02 12:49:18,044 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
java.io.IOException: Committed before 100 Continues
	at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
	at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: start
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: enter lock
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: now prepare
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: prepareCommit: flush
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]:   index before flush _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: startFullFlush
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: anyChanges? numDocsInRam=8 deletes=true hasTickets:false pendingChangesInFullFlush: false
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWFC][commitScheduler-15-thread-1]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 7 deleted terms (unique count=7) bytesUsed=1211, segment=_i8a5, aborted=false, numDocsInRAM=8, deleteQueue=DWDQ: [ generation: 5 ]]
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flush postings as segment _i8a5 numDocs=8
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 1 msec to write norms
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 1 msec to write docValues
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 0 msec to write points
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 0 msec to finish stored fields
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 8 msec to write postings and finish vectors
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: 0 msec to write fieldInfos
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: new segment has 0 deleted docs
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: new segment has no vectors; norms; docValues; prox; freqs
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flushedFiles=[_i8a5.nvd, _i8a5_Lucene50_0.tip, _i8a5_Lucene54_0.dvd, _i8a5_Lucene50_0.tim, _i8a5_Lucene54_0.dvm, _i8a5_Lucene50_0.doc, _i8a5.fdx, _i8a5.nvm, _i8a5_Lucene50_0.pos, _i8a5.fnm, _i8a5.fdt]
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flushed codec=Lucene60
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flushed: segment=_i8a5 ramUsed=0.282 MB newFlushedSize=0.033 MB docs/MB=243.064
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-15-thread-1]: flush time 12.739399 msec
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: publishFlushedSegment seg-private updates=null
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: publishFlushedSegment
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: push deletes  8 deleted terms (unique count=8) bytesUsed=1072 segmentPrivate?=false delGen=13 packetCount=1 totBytesUsed=1072
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: publish sets newSegment delGen=14 seg=_i8a5(6.0.1):C8
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: now checkpoint "_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8" [26 segments ; isCommit = false]
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: delete []
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: 0 msec to checkpoint
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: apply all deletes during flush
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: now apply all deletes for all segments maxDoc=1450172
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: open segment readers took 0 msec
2016-09-02 12:49:18,341 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: infos=[_5uop(6.0.1):C241991/704:delGen=698, _fb71(6.0.1):C389680/682:delGen=628, _haro(6.0.1):C485464/597:delGen=546, _hewu(6.0.1):c27085/185:delGen=168, _hjdo(6.0.1):c77072/249:delGen=225, _hpme(6.0.1):c57230/309:delGen=297, _hw4u(6.0.1):c24068/137:delGen=130, _i154(6.0.1):c68871/95:delGen=93, _i6m2(6.0.1):c38230/11:delGen=11, _i1z4(6.0.1):c9132/67:delGen=66, _i2wq(6.0.1):c7487/87:delGen=81, _i3wa(6.0.1):c6191/132:delGen=104, _i4ry(6.0.1):c6282/62:delGen=56, _i6ls(6.0.1):c1356/7:delGen=7, _i7ui(6.0.1):c7820/6:delGen=6, _i898(6.0.1):c263, _i88e(6.0.1):c256, _i8a2(6.0.1):c306, _i86g(6.0.1):c275, _i89i(6.0.1):c273, _i89s(6.0.1):c307, _i88y(6.0.1):c261, _i88o(6.0.1):c261, _i8a3(6.0.1):C1, _i8a4(6.0.1):C2, _i8a5(6.0.1):C8] packetCount=1
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyTermDeletes took 0.9 msec for 26 segments and 1 packets; 8 del terms visited; 109 seg terms visited
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: 0 new deleted documents
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes took 15 msec for 26 segments, 0 newly deleted docs (query deletes), 8 visited terms, allDeleted=null
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: prune sis=segments_hidp: _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8 minGen=15 packetCount=1
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: pruneDeletes: prune 1 packets; 0 packets remain
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: commitScheduler-15-thread-1 finishFullFlush success=true
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]: findMerges: 26 segments
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_haro(6.0.1):C485464/597:delGen=546 size=1165.550 MB
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_fb71(6.0.1):C389680/682:delGen=628 size=938.388 MB
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_5uop(6.0.1):C241991/704:delGen=698 size=580.037 MB
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_hjdo(6.0.1):c77072/249:delGen=225 size=185.438 MB
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i154(6.0.1):c68871/95:delGen=93 size=165.278 MB
2016-09-02 12:49:18,356 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_hpme(6.0.1):c57230/309:delGen=297 size=137.766 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i6m2(6.0.1):c38230/11:delGen=11 size=90.215 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_hewu(6.0.1):c27085/185:delGen=168 size=63.326 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_hw4u(6.0.1):c24068/137:delGen=130 size=62.890 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i1z4(6.0.1):c9132/67:delGen=66 size=20.958 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i7ui(6.0.1):c7820/6:delGen=6 size=20.457 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i2wq(6.0.1):c7487/87:delGen=81 size=18.896 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i3wa(6.0.1):c6191/132:delGen=104 size=15.820 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i4ry(6.0.1):c6282/62:delGen=56 size=14.736 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i6ls(6.0.1):c1356/7:delGen=7 size=3.616 MB
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i89i(6.0.1):c273 size=0.820 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i8a2(6.0.1):c306 size=0.803 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i89s(6.0.1):c307 size=0.801 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i88y(6.0.1):c261 size=0.797 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i86g(6.0.1):c275 size=0.794 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i898(6.0.1):c263 size=0.769 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i88e(6.0.1):c256 size=0.761 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i88o(6.0.1):c261 size=0.736 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i8a5(6.0.1):C8 size=0.033 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i8a4(6.0.1):C2 size=0.019 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   seg=_i8a3(6.0.1):C1 size=0.009 MB [floored]
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-15-thread-1]:   allowedSegmentCount=31 vs count=26 (eligible count=26) tooBigCount=0
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-15-thread-1]: now merge
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-15-thread-1]:   index: _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-15-thread-1]:   no more merges pending; now return
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: startCommit(): start
2016-09-02 12:49:18,372 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: startCommit index=_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8 changeCount=16
2016-09-02 12:49:18,388 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: startCommit: wrote pending segments file "pending_segments_hidq"
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: start
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: enter lock
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: now prepare
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: prepareCommit: flush
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]:   index before flush _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: startFullFlush
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: anyChanges? numDocsInRam=5 deletes=true hasTickets:false pendingChangesInFullFlush: false
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWFC][commitScheduler-12-thread-1]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0 4 deleted terms (unique count=4) bytesUsed=692, segment=_jtqc, aborted=false, numDocsInRAM=5, deleteQueue=DWDQ: [ generation: 92 ]]
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flush postings as segment _jtqc numDocs=5
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 0 msec to write norms
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 1 msec to write docValues
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 0 msec to write points
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 0 msec to finish stored fields
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 6 msec to write postings and finish vectors
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: 0 msec to write fieldInfos
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: new segment has 0 deleted docs
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: new segment has no vectors; norms; docValues; prox; freqs
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flushedFiles=[_jtqc_Lucene50_0.doc, _jtqc_Lucene50_0.tim, _jtqc.fdt, _jtqc.nvm, _jtqc_Lucene50_0.tip, _jtqc.fnm, _jtqc_Lucene50_0.pos, _jtqc.fdx, _jtqc_Lucene54_0.dvm, _jtqc.nvd, _jtqc_Lucene54_0.dvd]
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flushed codec=Lucene60
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flushed: segment=_jtqc ramUsed=0.258 MB newFlushedSize=0.013 MB docs/MB=379.479
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DWPT][commitScheduler-12-thread-1]: flush time 10.267016 msec
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: publishFlushedSegment seg-private updates=null
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: publishFlushedSegment
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: push deletes  5 deleted terms (unique count=5) bytesUsed=1072 segmentPrivate?=false delGen=184 packetCount=1 totBytesUsed=1072
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: publish sets newSegment delGen=185 seg=_jtqc(6.0.1):C5
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5" [26 segments ; isCommit = false]
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: delete []
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: 0 msec to checkpoint
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: apply all deletes during flush
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: now apply all deletes for all segments maxDoc=1450428
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: open segment readers took 0 msec
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: infos=[_5tw9(6.0.1):C241588/706:delGen=703, _fb82(6.0.1):C387586/788:delGen=727, _iin7(6.0.1):C491496/573:delGen=526, _i0at(6.0.1):c17859/119:delGen=112, _iy15(6.0.1):c32975/162:delGen=157, _iqm7(6.0.1):c67369/342:delGen=325, _jf5b(6.0.1):c22457/84:delGen=83, _j7ia(6.0.1):c65875/307:delGen=272, _jksn(6.0.1):c60239/75:delGen=70, _jthf(6.0.1):c27514/3:delGen=3, _jmdv(6.0.1):c7804/90:delGen=80, _jo5r(6.0.1):c7330/119:delGen=93, _jp6z(6.0.1):c7377/43:delGen=36, _jq41(6.0.1):c6791/32:delGen=31, _jquf(6.0.1):c5259/7:delGen=7, _jtpr(6.0.1):c119, _jtox(6.0.1):c108, _jtod(6.0.1):c97, _jtph(6.0.1):c114, _jtq1(6.0.1):c117, _jto3(6.0.1):c98, _jtp7(6.0.1):c123, _jton(6.0.1):c118, _jtqa(6.0.1):C5, _jtqb(6.0.1):C5, _jtqc(6.0.1):C5] packetCount=1
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyTermDeletes took 1.1 msec for 26 segments and 1 packets; 5 del terms visited; 77 seg terms visited
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: 0 new deleted documents
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes took 0 msec for 26 segments, 0 newly deleted docs (query deletes), 5 visited terms, allDeleted=null
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: prune sis=segments_htjo: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 minGen=186 packetCount=1
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: pruneDeletes: prune 1 packets; 0 packets remain
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: commitScheduler-12-thread-1 finishFullFlush success=true
2016-09-02 12:49:18,497 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]: findMerges: 26 segments
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_iin7(6.0.1):C491496/573:delGen=526 size=1181.402 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_fb82(6.0.1):C387586/788:delGen=727 size=933.087 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_5tw9(6.0.1):C241588/706:delGen=703 size=580.202 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_j7ia(6.0.1):c65875/307:delGen=272 size=162.085 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_iqm7(6.0.1):c67369/342:delGen=325 size=161.505 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jksn(6.0.1):c60239/75:delGen=70 size=144.827 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_iy15(6.0.1):c32975/162:delGen=157 size=78.643 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jthf(6.0.1):c27514/3:delGen=3 size=66.931 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jf5b(6.0.1):c22457/84:delGen=83 size=54.829 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_i0at(6.0.1):c17859/119:delGen=112 size=42.784 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jmdv(6.0.1):c7804/90:delGen=80 size=20.166 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jo5r(6.0.1):c7330/119:delGen=93 size=18.428 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jp6z(6.0.1):c7377/43:delGen=36 size=16.519 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jq41(6.0.1):c6791/32:delGen=31 size=15.395 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jquf(6.0.1):c5259/7:delGen=7 size=12.601 MB
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jton(6.0.1):c118 size=0.389 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtq1(6.0.1):c117 size=0.365 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtp7(6.0.1):c123 size=0.362 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtpr(6.0.1):c119 size=0.358 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtox(6.0.1):c108 size=0.335 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtph(6.0.1):c114 size=0.320 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jto3(6.0.1):c98 size=0.311 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtod(6.0.1):c97 size=0.287 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtqb(6.0.1):C5 size=0.021 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtqa(6.0.1):C5 size=0.019 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   seg=_jtqc(6.0.1):C5 size=0.014 MB [floored]
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [TMP][commitScheduler-12-thread-1]:   allowedSegmentCount=31 vs count=26 (eligible count=26) tooBigCount=0
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-12-thread-1]: now merge
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-12-thread-1]:   index: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [MS][commitScheduler-12-thread-1]:   no more merges pending; now return
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: startCommit(): start
2016-09-02 12:49:18,513 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: startCommit index=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 changeCount=10
2016-09-02 12:49:18,528 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: startCommit: wrote pending segments file "pending_segments_htjp"
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: done all syncs: [_i6ls.cfe, _i8a3.fdt, _i86g.cfs, _i8a3.fdx, _i89i.si, _i3wa.cfs, _haro.fdt, _i86g.si, _i89s.cfe, _haro_Lucene50_0.tim, _i6ls.cfs, _i8a5.nvd, _haro_Lucene50_0.tip, _i8a5_Lucene54_0.dvd, _5uop_je.liv, _haro.si, _i8a5_Lucene54_0.dvm, _i3wa.cfe, _i8a5.nvm, _i8a5.fnm, _i6ls_7.liv, _i89s.cfs, _i8a3.nvd, _i4ry_1k.liv, _i8a5.fdx, _i4ry.si, _i8a5.fdt, _i88y.cfe, _hjdo.si, _i7ui_6.liv, _haro.fdx, _i1z4.cfs, _i88e.si, _i8a5_Lucene50_0.pos, _i88y.cfs, _i8a3_Lucene50_0.doc, _i8a3.nvm, _i8a3.fnm, _i8a5.si, _fb71.nvd, _hpme.si, _i154.si, _i1z4.cfe, _i6m2.si, _i88o.si, _hpme.cfe, _fb71.nvm, _fb71.fnm, _5uop.si, _i8a3_Lucene50_0.pos, _i6ls.si, _i8a2.si, _i7ui.cfs, _fb71.fdx, _haro_Lucene54_0.dvm, _hjdo.cfs, _i154_2l.liv, _i7ui.cfe, _hjdo.cfe, _haro_Lucene54_0.dvd, _i88e.cfs, _fb71_Lucene50_0.tip, _i2wq.si, _fb71_Lucene50_0.tim, _i8a5_Lucene50_0.tip, _i8a5_Lucene50_0.tim, _i89s.si, _i8a2.cfs, _i898.si, _i88e.cfe, _hewu.si, _fb71_Lucene54_0.dvm, _i4ry.cfe, _i8a2.cfe, _i8a4_Lucene50_0.doc, _hw4u.si, _5uop_Lucene50_0.pos, _fb71_Lucene54_0.dvd, _i88y.si, _i8a3_Lucene50_0.tip, _fb71_hg.liv, _i88o.cfs, _i8a4.fdx, _fb71.fdt, _i8a3_Lucene50_0.tim, _i4ry.cfs, _i8a4.fdt, _i8a4.si, _5uop_Lucene54_0.dvd, _hewu.cfe, _i898.cfs, _5uop.nvd, _i8a4_Lucene50_0.pos, _5uop_Lucene50_0.doc, _5uop_Lucene54_0.dvm, _i898.cfe, _hewu_4o.liv, _i2wq_29.liv, _i8a4.nvm, _i8a4.fnm, _i3wa.si, _5uop.nvm, _5uop.fnm, _hewu.cfs, _haro_f6.liv, _i8a3_Lucene54_0.dvd, _i8a4.nvd, _i8a3_Lucene54_0.dvm, _hpme.cfs, _i3wa_2w.liv, _i2wq.cfe, _hw4u_3m.liv, _hpme_89.liv, _5uop_Lucene50_0.tip, _i8a4_Lucene54_0.dvd, _fb71_Lucene50_0.pos, _i1z4.si, _hjdo_69.liv, _5uop.fdx, _5uop_Lucene50_0.tim, _fb71.si, _hw4u.cfe, _i6m2.cfe, _i8a4_Lucene54_0.dvm, _5uop.fdt, _fb71_Lucene50_0.doc, _haro.nvd, _hw4u.cfs, _i8a5_Lucene50_0.doc, _haro.fnm, _haro.nvm, _i6m2.cfs, _i154.cfs, _i89i.cfs, _i8a4_Lucene50_0.tip, _i6m2_b.liv, _i88o.cfe, _haro_Lucene50_0.pos, _i89i.cfe, _i8a4_Lucene50_0.tim, _i154.cfe, _haro_Lucene50_0.doc, _i2wq.cfs, _i86g.cfe, _i8a3.si, _i1z4_1u.liv, _i7ui.si]
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: pendingCommit != null
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: done writing segments file "segments_hidq"
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: now checkpoint "_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8" [26 segments ; isCommit = true]
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.core.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
	commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard2_replica1\data\index.20160823134952477 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_hidp,generation=816973}
	commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard2_replica1\data\index.20160823134952477 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_hidq,generation=816974}
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.core.SolrDeletionPolicy newest commit generation = 816974
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: deleteCommits: now decRef commit "segments_hidp"
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: delete [segments_hidp]
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: 4 msec to checkpoint
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: delete []
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: took 388.6 msec
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: commit: done
2016-09-02 12:49:18,716 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: nrtIsCurrent: infoVersion matches: false; DW changes: false; BD changes: false
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: flush at getReader
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: startFullFlush
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: apply all deletes during flush
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: now apply all deletes for all segments maxDoc=1450172
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: open segment readers took 0 msec
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: applyDeletes: no segments; skipping
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-15-thread-1]: prune sis=segments_hidq: _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8 minGen=15 packetCount=0
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: incRefDeleter for NRT reader version=3357331 segments=_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: return reader version=3357331 reader=StandardDirectoryReader(segments_hidq:3357331:nrt _5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2 _i8a5(6.0.1):C8)
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-15-thread-1]: commitScheduler-15-thread-1 finishFullFlush success=true
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: getReader took 15 msec
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@4320bac9[transcribedReports_shard2_replica1] realtime]
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-15-thread-1]: delete []
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-15-thread-1]: decRefDeleter for NRT reader version=3357327 segments=_5uop(6.0.1):C241991/704:delGen=698 _fb71(6.0.1):C389680/682:delGen=628 _haro(6.0.1):C485464/597:delGen=546 _hewu(6.0.1):c27085/185:delGen=168 _hjdo(6.0.1):c77072/249:delGen=225 _hpme(6.0.1):c57230/309:delGen=297 _hw4u(6.0.1):c24068/137:delGen=130 _i154(6.0.1):c68871/95:delGen=93 _i6m2(6.0.1):c38230/11:delGen=11 _i1z4(6.0.1):c9132/67:delGen=66 _i2wq(6.0.1):c7487/87:delGen=81 _i3wa(6.0.1):c6191/132:delGen=104 _i4ry(6.0.1):c6282/62:delGen=56 _i6ls(6.0.1):c1356/7:delGen=7 _i7ui(6.0.1):c7820/6:delGen=6 _i898(6.0.1):c263 _i88e(6.0.1):c256 _i8a2(6.0.1):c306 _i86g(6.0.1):c275 _i89i(6.0.1):c273 _i89s(6.0.1):c307 _i88y(6.0.1):c261 _i88o(6.0.1):c261 _i8a3(6.0.1):C1 _i8a4(6.0.1):C2
2016-09-02 12:49:18,731 level=INFO    thread=commitScheduler-15-thread-1 category=org.apache.solr.update.DirectUpdateHandler2 end_commit_flush
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: done all syncs: [_jtqb.fdt, _jto3.cfe, _jksn.cfs, _iy15.cfs, _jtod.cfe, _jtqb.fdx, _jtqa_Lucene50_0.tim, _fb82.fdx, _jtox.cfe, _jksn_1y.liv, _jmdv.si, _iy15_4d.liv, _jto3.cfs, _iqm7.cfs, _5tw9_Lucene50_0.pos, _iin7.fdt, _jksn.cfe, _jq41.cfe, _jtqb_Lucene50_0.pos, _jtqa_Lucene54_0.dvm, _iin7.fdx, _jf5b.cfe, _iin7_Lucene50_0.pos, _iy15.cfe, _jf5b.si, _iqm7.si, _jtq1.cfs, _iqm7.cfe, _jtox.cfs, _jtox.si, _fb82_Lucene54_0.dvm, _jf5b.cfs, _jq41.cfs, _fb82_Lucene54_0.dvd, _jtqb.nvd, _fb82_Lucene50_0.pos, _iin7.si, _jf5b_2b.liv, _fb82_Lucene50_0.doc, _jtqb.si, _jtqb.nvm, _jtqb.fnm, _5tw9_Lucene50_0.doc, _iin7.nvd, _jq41_v.liv, _fb82.fdt, _iin7.nvm, _iin7.fnm, _jtph.cfs, _jquf_7.liv, _jthf.si, _5tw9_Lucene54_0.dvm, _jthf_3.liv, _jtpr.si, _5tw9_Lucene54_0.dvd, _fb82_Lucene50_0.tim, _5tw9_Lucene50_0.tim, _jtqc.si, _fb82_Lucene50_0.tip, _5tw9_Lucene50_0.tip, _i0at.si, _jton.si, _fb82.nvd, _jquf.cfs, _fb82.nvm, _fb82.fnm, _iin7_em.liv, _jtqa_Lucene54_0.dvd, _iin7_Lucene50_0.doc, _j7ia.si, _iqm7_91.liv, _jtph.cfe, _jquf.cfe, _jtqb_Lucene50_0.doc, _jtqc_Lucene54_0.dvm, _jtqa_Lucene50_0.tip, _jton.cfe, _jo5r.cfs, _fb82_k7.liv, _5tw9.fdt, _jthf.cfs, _jton.cfs, _j7ia.cfe, _jmdv.cfe, _5tw9.fdx, _i0at_34.liv, _jtp7.si, _jtqc_Lucene54_0.dvd, _jp6z.si, _jtqa.si, _jtqc.fdt, _j7ia_7k.liv, _jmdv.cfs, _jtqc.fdx, _jto3.si, _jtqc_Lucene50_0.tim, _jtqc.nvm, _jtqc_Lucene50_0.tip, _jtqc.fnm, _iy15.si, _5tw9.si, _jtqc.nvd, _jthf.cfe, _jtqc_Lucene50_0.doc, _jtp7.cfs, _jtqa_Lucene50_0.pos, _jtqa.nvm, _jtqa.fnm, _i0at.cfe, _jtqb_Lucene54_0.dvm, _jmdv_28.liv, _jo5r.si, _iin7_Lucene50_0.tip, _i0at.cfs, _jo5r_2l.liv, _iin7_Lucene50_0.tim, _jp6z.cfe, _jtqa.nvd, _jtp7.cfe, _5tw9_jj.liv, _jtqb_Lucene50_0.tim, _jtpr.cfs, _iin7_Lucene54_0.dvm, _jp6z.cfs, _jtqc_Lucene50_0.pos, _jtod.si, _jtqa_Lucene50_0.doc, _jp6z_10.liv, _jq41.si, _jtqb_Lucene50_0.tip, _jksn.si, _jtq1.cfe, _5tw9.nvd, _jtpr.cfe, _jtqa.fdx, _jtph.si, _j7ia.cfs, _jtqa.fdt, _jo5r.cfe, _jtod.cfs, _5tw9.nvm, _5tw9.fnm, _jtqb_Lucene54_0.dvd, _jquf.si, _iin7_Lucene54_0.dvd, _fb82.si, _jtq1.si]
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: pendingCommit != null
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: done writing segments file "segments_htjp"
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5" [26 segments ; isCommit = true]
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.core.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
	commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard1_replica1\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_htjo,generation=831444}
	commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard1_replica1\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_htjp,generation=831445}
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.core.SolrDeletionPolicy newest commit generation = 831445
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: deleteCommits: now decRef commit "segments_htjo"
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: delete [segments_htjo]
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: 0 msec to checkpoint
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: delete []
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: took 435.8 msec
2016-09-02 12:49:18,919 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: commit: done
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: nrtIsCurrent: infoVersion matches: false; DW changes: false; BD changes: false
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: flush at getReader
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: startFullFlush
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: apply all deletes during flush
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: now apply all deletes for all segments maxDoc=1450428
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: open segment readers took 0 msec
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: applyDeletes: no segments; skipping
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [BD][commitScheduler-12-thread-1]: prune sis=segments_htjp: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 minGen=186 packetCount=0
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: incRefDeleter for NRT reader version=3536056 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: return reader version=3536056 reader=StandardDirectoryReader(segments_htjp:3536056:nrt _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5)
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [DW][commitScheduler-12-thread-1]: commitScheduler-12-thread-1 finishFullFlush success=true
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: getReader took 0 msec
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@3b4ec22[transcribedReports_shard1_replica1] realtime]
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IFD][commitScheduler-12-thread-1]: delete []
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.LoggingInfoStream [IW][commitScheduler-12-thread-1]: decRefDeleter for NRT reader version=3536052 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5
2016-09-02 12:49:18,935 level=INFO    thread=commitScheduler-12-thread-1 category=org.apache.solr.update.DirectUpdateHandler2 end_commit_flush
2016-09-02 12:49:19,106 level=INFO    thread=qtp762152757-19 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50 
2016-09-02 12:49:19,106 level=INFO    thread=qtp762152757-19 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2 
2016-09-02 12:49:19,106 level=INFO    thread=qtp762152757-19 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
java.io.IOException: Committed before 100 Continues
	at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
	at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:19,153 level=INFO    thread=qtp762152757-18 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711925396]} 0 37
2016-09-02 12:49:19,169 level=INFO    thread=qtp762152757-20 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50 
2016-09-02 12:49:19,169 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2 
2016-09-02 12:49:19,169 level=INFO    thread=qtp762152757-20 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
java.io.IOException: Committed before 100 Continues
	at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
	at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:19,919 level=INFO    thread=qtp762152757-20 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol004:8983/solr/transcribedReports_shard3_replica4/&wt=javabin&version=2}{add=[711869839 (1544364291420848128)]} 0 17
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: anyChanges? numDocsInRam=1 deletes=true hasTickets:false pendingChangesInFullFlush: false
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: nrtIsCurrent: infoVersion matches: false; DW changes: true; BD changes: false
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: flush at getReader
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: startFullFlush
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: anyChanges? numDocsInRam=1 deletes=true hasTickets:false pendingChangesInFullFlush: false
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWFC][qtp762152757-17]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_jtqd, aborted=false, numDocsInRAM=1, deleteQueue=DWDQ: [ generation: 94 ]]
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flush postings as segment _jtqd numDocs=1
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 1 msec to write norms
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 0 msec to write docValues
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 0 msec to write points
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 2 msec to finish stored fields
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 2 msec to write postings and finish vectors
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: 0 msec to write fieldInfos
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: new segment has 0 deleted docs
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: new segment has no vectors; norms; docValues; prox; freqs
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flushedFiles=[_jtqd_Lucene50_0.tip, _jtqd.nvm, _jtqd_Lucene54_0.dvm, _jtqd_Lucene50_0.tim, _jtqd_Lucene50_0.doc, _jtqd.fdx, _jtqd_Lucene50_0.pos, _jtqd.nvd, _jtqd.fdt, _jtqd.fnm, _jtqd_Lucene54_0.dvd]
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flushed codec=Lucene60
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flushed: segment=_jtqd ramUsed=0.242 MB newFlushedSize=0.004 MB docs/MB=253.402
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DWPT][qtp762152757-17]: flush time 8.600604 msec
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: publishFlushedSegment seg-private updates=null
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: publishFlushedSegment
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: push deletes  1 deleted terms (unique count=1) bytesUsed=1072 segmentPrivate?=false delGen=188 packetCount=1 totBytesUsed=1072
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: publish sets newSegment delGen=189 seg=_jtqd(6.0.1):C1
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1" [27 segments ; isCommit = false]
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete []
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: 0 msec to checkpoint
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: apply all deletes during flush
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: now apply all deletes for all segments maxDoc=1450429
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: open segment readers took 0 msec
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: infos=[_5tw9(6.0.1):C241588/706:delGen=703, _fb82(6.0.1):C387586/788:delGen=727, _iin7(6.0.1):C491496/573:delGen=526, _i0at(6.0.1):c17859/119:delGen=112, _iy15(6.0.1):c32975/162:delGen=157, _iqm7(6.0.1):c67369/342:delGen=325, _jf5b(6.0.1):c22457/84:delGen=83, _j7ia(6.0.1):c65875/307:delGen=272, _jksn(6.0.1):c60239/75:delGen=70, _jthf(6.0.1):c27514/3:delGen=3, _jmdv(6.0.1):c7804/90:delGen=80, _jo5r(6.0.1):c7330/119:delGen=93, _jp6z(6.0.1):c7377/43:delGen=36, _jq41(6.0.1):c6791/32:delGen=31, _jquf(6.0.1):c5259/7:delGen=7, _jtpr(6.0.1):c119, _jtox(6.0.1):c108, _jtod(6.0.1):c97, _jtph(6.0.1):c114, _jtq1(6.0.1):c117, _jto3(6.0.1):c98, _jtp7(6.0.1):c123, _jton(6.0.1):c118, _jtqa(6.0.1):C5, _jtqb(6.0.1):C5, _jtqc(6.0.1):C5, _jtqd(6.0.1):C1] packetCount=1
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyTermDeletes took 1.1 msec for 27 segments and 1 packets; 1 del terms visited; 28 seg terms visited
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: 0 new deleted documents
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes took 0 msec for 27 segments, 0 newly deleted docs (query deletes), 1 visited terms, allDeleted=null
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: prune sis=segments_htjp: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1 minGen=190 packetCount=1
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: pruneDeletes: prune 1 packets; 0 packets remain
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: incRefDeleter for NRT reader version=3536058 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: return reader version=3536058 reader=StandardDirectoryReader(segments_htjp:3536058:nrt _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1)
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: qtp762152757-17 finishFullFlush success=true
2016-09-02 12:49:19,997 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]: findMerges: 27 segments
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_iin7(6.0.1):C491496/573:delGen=526 size=1181.402 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_fb82(6.0.1):C387586/788:delGen=727 size=933.087 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_5tw9(6.0.1):C241588/706:delGen=703 size=580.202 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_j7ia(6.0.1):c65875/307:delGen=272 size=162.085 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_iqm7(6.0.1):c67369/342:delGen=325 size=161.505 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jksn(6.0.1):c60239/75:delGen=70 size=144.827 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_iy15(6.0.1):c32975/162:delGen=157 size=78.643 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jthf(6.0.1):c27514/3:delGen=3 size=66.931 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jf5b(6.0.1):c22457/84:delGen=83 size=54.829 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_i0at(6.0.1):c17859/119:delGen=112 size=42.784 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jmdv(6.0.1):c7804/90:delGen=80 size=20.166 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jo5r(6.0.1):c7330/119:delGen=93 size=18.428 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jp6z(6.0.1):c7377/43:delGen=36 size=16.519 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jq41(6.0.1):c6791/32:delGen=31 size=15.395 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jquf(6.0.1):c5259/7:delGen=7 size=12.601 MB
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jton(6.0.1):c118 size=0.389 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtq1(6.0.1):c117 size=0.365 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtp7(6.0.1):c123 size=0.362 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtpr(6.0.1):c119 size=0.358 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtox(6.0.1):c108 size=0.335 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtph(6.0.1):c114 size=0.320 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jto3(6.0.1):c98 size=0.311 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtod(6.0.1):c97 size=0.287 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtqb(6.0.1):C5 size=0.021 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtqa(6.0.1):C5 size=0.019 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtqc(6.0.1):C5 size=0.014 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   seg=_jtqd(6.0.1):C1 size=0.004 MB [floored]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [TMP][qtp762152757-17]:   allowedSegmentCount=31 vs count=27 (eligible count=27) tooBigCount=0
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [MS][qtp762152757-17]: now merge
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [MS][qtp762152757-17]:   index: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [MS][qtp762152757-17]:   no more merges pending; now return
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: getReader took 16 msec
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@4bffc6cf[transcribedReports_shard1_replica1] realtime]
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete []
2016-09-02 12:49:20,013 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: decRefDeleter for NRT reader version=3536056 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5
2016-09-02 12:49:20,060 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.IndexFingerprint IndexFingerprint millis:42.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=1544364291420848128, maxInHash=1544364291420848128, versionsHash=5743601267512047892, numVersions=1446979, numDocs=1446979, maxDoc=1450429}
2016-09-02 12:49:20,060 level=INFO    thread=qtp762152757-17 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=true&getVersions=100&wt=javabin&version=2} status=0 QTime=65
2016-09-02 12:49:20,060 level=INFO    thread=qtp762152757-16 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://atlprwfftsol002:8983/solr/transcribedReports_shard2_replica2/&wt=javabin&version=2}{add=[711880061 (1544364291558211584)]} 0 7
2016-09-02 12:49:20,075 level=INFO    thread=qtp762152757-19 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/get params={getUpdates=-1544364282917945344&distrib=false&qt=/get&onlyIfActive=false&wt=javabin&version=2} status=0 QTime=1
2016-09-02 12:49:20,075 level=INFO    thread=qtp762152757-19 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
2016-09-02 12:49:20,075 level=INFO    thread=qtp762152757-19 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 2
2016-09-02 12:49:20,106 level=INFO    thread=qtp762152757-13 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{add=[711880746]} 0 58
2016-09-02 12:49:20,106 level=INFO    thread=qtp762152757-21 category=org.apache.solr.security.RuleBasedAuthorizationPlugin request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@2fd51e50 
2016-09-02 12:49:20,106 level=INFO    thread=qtp762152757-21 category=org.apache.solr.servlet.HttpSolrCall USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [transcribedReports,], Path: [/update] path : /update params :version=2.2 
2016-09-02 12:49:20,106 level=INFO    thread=qtp762152757-21 category=org.apache.solr.servlet.SolrDispatchFilter Could not consume full client request
java.io.IOException: Committed before 100 Continues
	at org.eclipse.jetty.server.HttpChannelOverHttp.continue100(HttpChannelOverHttp.java:206)
	at org.eclipse.jetty.server.Request.getInputStream(Request.java:802)
	at org.apache.solr.servlet.SolrDispatchFilter.consumeInputFully(SolrDispatchFilter.java:255)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:246)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Unknown Source)
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: start
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: enter lock
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: now prepare
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: prepareCommit: flush
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]:   index before flush _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: startFullFlush
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: apply all deletes during flush
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: now apply all deletes for all segments maxDoc=1450429
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: open segment readers took 0 msec
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: no segments; skipping
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: prune sis=segments_htjp: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1 minGen=190 packetCount=0
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: qtp762152757-17 finishFullFlush success=true
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: startCommit(): start
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: startCommit index=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1 changeCount=14
2016-09-02 12:49:20,153 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: startCommit: wrote pending segments file "pending_segments_htjq"
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: done all syncs: [_jtqb.fdt, _jto3.cfe, _jksn.cfs, _iy15.cfs, _jtod.cfe, _jtqb.fdx, _jtqa_Lucene50_0.tim, _fb82.fdx, _jtox.cfe, _jtqd.nvd, _jksn_1y.liv, _jmdv.si, _iy15_4d.liv, _jto3.cfs, _iqm7.cfs, _5tw9_Lucene50_0.pos, _iin7.fdt, _jksn.cfe, _jq41.cfe, _jtqb_Lucene50_0.pos, _jtqd.nvm, _jtqa_Lucene54_0.dvm, _iin7.fdx, _jf5b.cfe, _iin7_Lucene50_0.pos, _iy15.cfe, _jf5b.si, _jtqd.fdx, _iqm7.si, _jtq1.cfs, _iqm7.cfe, _jtox.cfs, _jtox.si, _jtqd.fdt, _fb82_Lucene54_0.dvm, _jf5b.cfs, _jq41.cfs, _fb82_Lucene54_0.dvd, _jtqb.nvd, _fb82_Lucene50_0.pos, _jtqd.fnm, _iin7.si, _jf5b_2b.liv, _fb82_Lucene50_0.doc, _jtqd_Lucene50_0.tip, _jtqb.si, _jtqb.nvm, _jtqb.fnm, _5tw9_Lucene50_0.doc, _jtqd_Lucene50_0.tim, _iin7.nvd, _jq41_v.liv, _fb82.fdt, _iin7.nvm, _iin7.fnm, _jtph.cfs, _jquf_7.liv, _jthf.si, _5tw9_Lucene54_0.dvm, _jthf_3.liv, _jtpr.si, _5tw9_Lucene54_0.dvd, _fb82_Lucene50_0.tim, _5tw9_Lucene50_0.tim, _jtqc.si, _fb82_Lucene50_0.tip, _5tw9_Lucene50_0.tip, _i0at.si, _jton.si, _fb82.nvd, _jquf.cfs, _fb82.nvm, _fb82.fnm, _iin7_em.liv, _jtqa_Lucene54_0.dvd, _iin7_Lucene50_0.doc, _j7ia.si, _iqm7_91.liv, _jtph.cfe, _jquf.cfe, _jtqb_Lucene50_0.doc, _jtqc_Lucene54_0.dvm, _jtqa_Lucene50_0.tip, _jtqd_Lucene50_0.pos, _jton.cfe, _jo5r.cfs, _fb82_k7.liv, _5tw9.fdt, _jthf.cfs, _jton.cfs, _j7ia.cfe, _jmdv.cfe, _5tw9.fdx, _i0at_34.liv, _jtp7.si, _jtqc_Lucene54_0.dvd, _jp6z.si, _jtqa.si, _jtqc.fdt, _j7ia_7k.liv, _jmdv.cfs, _jtqc.fdx, _jtqd_Lucene50_0.doc, _jto3.si, _jtqc_Lucene50_0.tim, _jtqc.nvm, _jtqc_Lucene50_0.tip, _jtqc.fnm, _iy15.si, _5tw9.si, _jtqc.nvd, _jthf.cfe, _jtqc_Lucene50_0.doc, _jtp7.cfs, _jtqa_Lucene50_0.pos, _jtqa.nvm, _jtqa.fnm, _i0at.cfe, _jtqb_Lucene54_0.dvm, _jmdv_28.liv, _jo5r.si, _iin7_Lucene50_0.tip, _i0at.cfs, _jo5r_2l.liv, _iin7_Lucene50_0.tim, _jp6z.cfe, _jtqa.nvd, _jtp7.cfe, _5tw9_jj.liv, _jtqb_Lucene50_0.tim, _jtpr.cfs, _jtqd_Lucene54_0.dvd, _iin7_Lucene54_0.dvm, _jp6z.cfs, _jtqc_Lucene50_0.pos, _jtqd_Lucene54_0.dvm, _jtod.si, _jtqa_Lucene50_0.doc, _jp6z_10.liv, _jq41.si, _jtqb_Lucene50_0.tip, _jksn.si, _jtq1.cfe, _5tw9.nvd, _jtpr.cfe, _jtqa.fdx, _jtph.si, _j7ia.cfs, _jtqa.fdt, _jtqd.si, _jo5r.cfe, _jtod.cfs, _5tw9.nvm, _5tw9.fnm, _jtqb_Lucene54_0.dvd, _jquf.si, _iin7_Lucene54_0.dvd, _fb82.si, _jtq1.si]
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: pendingCommit != null
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: done writing segments file "segments_htjq"
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: now checkpoint "_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1" [27 segments ; isCommit = true]
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.core.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
	commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard1_replica1\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_htjp,generation=831445}
	commit{dir=NRTCachingDirectory(MMapDirectory@E:\solr_home\transcribedReports_shard1_replica1\data\index lockFactory=org.apache.lucene.store.NativeFSLockFactory@2998ad49; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_htjq,generation=831446}
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.core.SolrDeletionPolicy newest commit generation = 831446
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: deleteCommits: now decRef commit "segments_htjp"
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete [segments_htjp]
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: 0 msec to checkpoint
2016-09-02 12:49:20,356 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete []
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: took 211.5 msec
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: commit: done
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: nrtIsCurrent: infoVersion matches: false; DW changes: false; BD changes: false
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: flush at getReader
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: startFullFlush
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: apply all deletes during flush
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: now apply all deletes for all segments maxDoc=1450429
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: open segment readers took 0 msec
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: applyDeletes: no segments; skipping
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [BD][qtp762152757-17]: prune sis=segments_htjq: _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1 minGen=190 packetCount=0
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: incRefDeleter for NRT reader version=3536060 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: return reader version=3536060 reader=StandardDirectoryReader(segments_htjq:3536060:nrt _5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1)
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [DW][qtp762152757-17]: qtp762152757-17 finishFullFlush success=true
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: getReader took 0 msec
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.search.SolrIndexSearcher Opening [Searcher@11297ec7[transcribedReports_shard1_replica1] realtime]
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IFD][qtp762152757-17]: delete []
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.LoggingInfoStream [IW][qtp762152757-17]: decRefDeleter for NRT reader version=3536058 segments=_5tw9(6.0.1):C241588/706:delGen=703 _fb82(6.0.1):C387586/788:delGen=727 _iin7(6.0.1):C491496/573:delGen=526 _i0at(6.0.1):c17859/119:delGen=112 _iy15(6.0.1):c32975/162:delGen=157 _iqm7(6.0.1):c67369/342:delGen=325 _jf5b(6.0.1):c22457/84:delGen=83 _j7ia(6.0.1):c65875/307:delGen=272 _jksn(6.0.1):c60239/75:delGen=70 _jthf(6.0.1):c27514/3:delGen=3 _jmdv(6.0.1):c7804/90:delGen=80 _jo5r(6.0.1):c7330/119:delGen=93 _jp6z(6.0.1):c7377/43:delGen=36 _jq41(6.0.1):c6791/32:delGen=31 _jquf(6.0.1):c5259/7:delGen=7 _jtpr(6.0.1):c119 _jtox(6.0.1):c108 _jtod(6.0.1):c97 _jtph(6.0.1):c114 _jtq1(6.0.1):c117 _jto3(6.0.1):c98 _jtp7(6.0.1):c123 _jton(6.0.1):c118 _jtqa(6.0.1):C5 _jtqb(6.0.1):C5 _jtqc(6.0.1):C5 _jtqd(6.0.1):C1
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.DirectUpdateHandler2 end_commit_flush
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-17 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&commit_end_point=true&wt=javabin&version=2}{commit=} 0 217
2016-09-02 12:49:20,372 level=INFO    thread=qtp762152757-19 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.handler.admin.CollectionsHandler Invoked Collection Action :clusterstatus with params action=clusterstatus&wt=json and sendToOCPQueue=true
2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.common.cloud.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (4)
2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.common.cloud.ZkStateReader Load collection config from: [/collections/transcribedReports]
2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.common.cloud.ZkStateReader path=[/collections/transcribedReports] [configName]=[transcribedReports] specified config exists in ZooKeeper
2016-09-02 12:49:20,388 level=INFO    thread=qtp762152757-18 category=org.apache.solr.servlet.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=clusterstatus&wt=json} status=0 QTime=6
2016-09-02 12:49:20,450 level=INFO    thread=qtp762152757-12 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/replication params={generation=831446&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=57
2016-09-02 12:49:20,544 level=INFO    thread=qtp762152757-15 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/replication params={generation=831446&qt=/replication&file=_5tw9_Lucene54_0.dvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
2016-09-02 12:49:20,560 level=INFO    thread=qtp762152757-17 category=org.apache.solr.core.SolrCore.Request [transcribedReports_shard1_replica1]  webapp=/solr path=/replication params={generation=831446&qt=/replication&file=_5tw9.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
2016-09-02 12:49:20,669 level=INFO    thread=qtp762152757-16 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard1_replica1]  webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://atlprwfftsol003:8983/solr/transcribedReports_shard3_replica3/&wt=javabin&version=2}{add=[711926412 (1544364292191551488)]} 0 33
2016-09-02 12:49:20,685 level=INFO    thread=qtp762152757-13 category=org.apache.solr.update.processor.IgnoreCommitOptimizeUpdateProcessorFactory commit from client application ignored with status code: 200
2016-09-02 12:49:20,685 level=INFO    thread=qtp762152757-13 category=org.apache.solr.update.processor.LogUpdateProcessorFactory [transcribedReports_shard2_replica1]  webapp=/solr path=/update params={version=2.2}{} 0 2

2/ clusterstate.json

    "solr_cluster_status.json.cluster": {
        "collections": {
            "transcribedReports": {
                "autoAddReplicas": "false",
                "configName": "transcribedReports",
                "maxShardsPerNode": "4",
                "replicationFactor": "3",
                "router": {
                    "name": "compositeId"
                },
                "shards": {
                    "shard1": {
                        "range": "80000000-bfffffff",
                        "replicas": {
                            "core_node13": {
                                "base_url": "http://atlprwfftsol001:8983/solr",
                                "core": "transcribedReports_shard1_replica1",
                                "leader": "true",
                                "node_name": "atlprwfftsol001:8983_solr",
                                "state": "active"
                            },
                            "core_node14": {
                                "base_url": "http://atlprwfftsol002:8983/solr",
                                "core": "transcribedReports_shard1_replica2",
                                "node_name": "atlprwfftsol002:8983_solr",
                                "state": "down"
                            }
                        },
                        "state": "active"
                    },
                    "shard2": {
                        "range": "c0000000-ffffffff",
                        "replicas": {
                            "core_node4": {
                                "base_url": "http://atlprwfftsol001:8983/solr",
                                "core": "transcribedReports_shard2_replica1",
                                "node_name": "atlprwfftsol001:8983_solr",
                                "state": "active"
                            },
                            "core_node8": {
                                "base_url": "http://atlprwfftsol002:8983/solr",
                                "core": "transcribedReports_shard2_replica2",
                                "leader": "true",
                                "node_name": "atlprwfftsol002:8983_solr",
                                "state": "active"
                            }
                        },
                        "state": "active"
                    },
                    "shard3": {
                        "range": "0-3fffffff",
                        "replicas": {
                            "core_node10": {
                                "base_url": "http://atlprwfftsol003:8983/solr",
                                "core": "transcribedReports_shard3_replica3",
                                "leader": "true",
                                "node_name": "atlprwfftsol003:8983_solr",
                                "state": "active"
                            },
                            "core_node12": {
                                "base_url": "http://atlprwfftsol004:8983/solr",
                                "core": "transcribedReports_shard3_replica4",
                                "node_name": "atlprwfftsol004:8983_solr",
                                "state": "active"
                            }
                        },
                        "state": "active"
                    },
                    "shard4": {
                        "range": "40000000-7fffffff",
                        "replicas": {
                            "core_node11": {
                                "base_url": "http://atlprwfftsol003:8983/solr",
                                "core": "transcribedReports_shard4_replica4",
                                "leader": "true",
                                "node_name": "atlprwfftsol003:8983_solr",
                                "state": "active"
                            },
                            "core_node9": {
                                "base_url": "http://atlprwfftsol004:8983/solr",
                                "core": "transcribedReports_shard4_replica3",
                                "node_name": "atlprwfftsol004:8983_solr",
                                "state": "active"
                            }
                        },
                        "state": "active"
                    }
                },
                "znodeVersion": 178208
            }
        },
        "live_nodes": [
            "atlprwfftsol002:8983_solr",
            "atlprwfftsol004:8983_solr",
            "atlprwfftsol003:8983_solr",
            "atlprwfftsol001:8983_solr"
        ]
    }


3/ transcribedReports cluster's solrconfig.xml.  I think the only changes to default are for the updateRequestProcessorChain (ignore commits from clients and auto delete expired documents), and auto commit configuration hard autocommit 15 seconds, opensearch=false soft autocommit every 15 *minutes*)

    <updateRequestProcessorChain name="FFTtranscribedReportsProcessorChain" default="true">
        <processor class="solr.IgnoreCommitOptimizeUpdateProcessorFactory">
          <int name="statusCode">200</int>
        </processor>
        <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
            <int name="autoDeletePeriodSeconds">30</int>
            <str name="ttlFieldName">_ttl_</str>
            <str name="expirationFieldName">expiryDate</str>
        </processor>
        <processor class="solr.FirstFieldValueUpdateProcessorFactory">
            <str name="fieldName">expiryDate</str>
        </processor>
        <processor class="solr.LogUpdateProcessorFactory" />
        <processor class="solr.DistributedUpdateProcessorFactory" />
        <processor class="solr.RunUpdateProcessorFactory" />
    </updateRequestProcessorChain>


        <autoCommit>
            <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
            <openSearcher>false</openSearcher>
        </autoCommit>

        <!-- setting softAutoCommit to 15 minutes for fft transcribedReports usage. -->
        <autoSoftCommit>
            <maxTime>${solr.autoSoftCommit.maxTime:900000}</maxTime>
        </autoSoftCommit>

(full solrconfig.xml for this collection below)


<?xml version="1.0" encoding="UTF-8" ?>
<!--
 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements.  See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-->

<!--
     For more details about configurations options that may appear in
     this file, see http://wiki.apache.org/solr/SolrConfigXml.
-->
<config>

    <!-- Having this in the config means that Solr will honour schema.xml rather
         than constructing its own schema (which it calls "managed-schema")
         This means we can manage the schema using the zkCli script in solr and
         keep all the config under version control.
    -->
    <schemaFactory class="ClassicIndexSchemaFactory"/>

    <!-- In all configuration below, a prefix of "solr." for class names
         is an alias that causes solr to search appropriate packages,
         including org.apache.solr.(search|update|request|core|analysis)

         You may also specify a fully qualified Java classname if you
         have your own custom plugins.
      -->

    <!-- Controls what version of Lucene various components of Solr
         adhere to.  Generally, you want to use the latest version to
         get all bug fixes and improvements. It is highly recommended
         that you fully re-index after changing this setting as it can
         affect both how text is indexed and queried.
    -->
    <luceneMatchVersion>6.0.0</luceneMatchVersion>

    <!-- Data Directory

         Used to specify an alternate directory to hold all index data
         other than the default ./data under the Solr home.  If
         replication is in use, this should match the replication
         configuration.
      -->
    <dataDir>${solr.data.dir:}</dataDir>


    <!-- The DirectoryFactory to use for indexes.

         solr.StandardDirectoryFactory is filesystem
         based and tries to pick the best implementation for the current
         JVM and platform.  solr.NRTCachingDirectoryFactory, the default,
         wraps solr.StandardDirectoryFactory and caches small files in memory
         for better NRT performance.

         One can force a particular implementation via solr.MMapDirectoryFactory,
         solr.NIOFSDirectoryFactory, or solr.SimpleFSDirectoryFactory.

         solr.RAMDirectoryFactory is memory based, not
         persistent, and doesn't work with replication.
      -->
    <directoryFactory name="DirectoryFactory"
                      class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}">
    </directoryFactory>

    <!-- The CodecFactory for defining the format of the inverted index.
         The default implementation is SchemaCodecFactory, which is the official Lucene
         index format, but hooks into the schema to provide per-field customization of
         the postings lists and per-document values in the fieldType element
         (postingsFormat/docValuesFormat). Note that most of the alternative implementations
         are experimental, so if you choose to customize the index format, it's a good
         idea to convert back to the official format e.g. via IndexWriter.addIndexes(IndexReader)
         before upgrading to a newer version to avoid unnecessary reindexing.
         A "compressionMode" string element can be added to <codecFactory> to choose
         between the existing compression modes in the default codec: "BEST_SPEED" (default)
         or "BEST_COMPRESSION".
    -->
    <codecFactory class="solr.SchemaCodecFactory"/>

    <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         Index Config - These settings control low-level behavior of indexing
         Most example settings here show the default value, but are commented
         out, to more easily see where customizations have been made.

         Note: This replaces <indexDefaults> and <mainIndex> from older versions
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
    <indexConfig>

        <!-- LockFactory

             This option specifies which Lucene LockFactory implementation
             to use.

             single = SingleInstanceLockFactory - suggested for a
                      read-only index or when there is no possibility of
                      another process trying to modify the index.
             native = NativeFSLockFactory - uses OS native file locking.
                      Do not use when multiple solr webapps in the same
                      JVM are attempting to share a single index.
             simple = SimpleFSLockFactory  - uses a plain file for locking

             Defaults: 'native' is default for Solr3.6 and later, otherwise
                       'simple' is the default

             More details on the nuances of each LockFactory...
             http://wiki.apache.org/lucene-java/AvailableLockFactories
        -->
        <lockType>${solr.lock.type:native}</lockType>

        <!-- Lucene Infostream

             To aid in advanced debugging, Lucene provides an "InfoStream"
             of detailed information when indexing.

             Setting the value to true will instruct the underlying Lucene
             IndexWriter to write its info stream to solr's log. By default,
             this is enabled here, and controlled through log4j.properties.
          -->
        <infoStream>true</infoStream>
    </indexConfig>


    <!-- JMX

         This example enables JMX if and only if an existing MBeanServer
         is found, use this if you want to configure JMX through JVM
         parameters. Remove this to disable exposing Solr configuration
         and statistics to JMX.

         For more details see http://wiki.apache.org/solr/SolrJmx
      -->
    <jmx/>
    <!-- If you want to connect to a particular server, specify the
         agentId
      -->
    <!-- <jmx agentId="myAgent" /> -->
    <!-- If you want to start a new MBeanServer, specify the serviceUrl -->
    <!-- <jmx serviceUrl="service:jmx:rmi:///jndi/rmi://localhost:9999/solr"/>
      -->

    <!-- The default high-performance update handler -->
    <updateHandler class="solr.DirectUpdateHandler2">

        <!-- Enables a transaction log, used for real-time get, durability, and
             and solr cloud replica recovery.  The log can grow as big as
             uncommitted changes to the index, so use of a hard autoCommit
             is recommended (see below).
             "dir" - the target directory for transaction logs, defaults to the
                    solr data directory.
             "numVersionBuckets" - sets the number of buckets used to keep
                    track of max version values when checking for re-ordered
                    updates; increase this value to reduce the cost of
                    synchronizing access to version buckets during high-volume
                    indexing, this requires 8 bytes (long) * numVersionBuckets
                    of heap space per Solr core.
        -->
        <updateLog>
            <str name="dir">${solr.ulog.dir:}</str>
            <int name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}</int>
        </updateLog>

        <!-- AutoCommit

             Perform a hard commit automatically under certain conditions.
             Instead of enabling autoCommit, consider using "commitWithin"
             when adding documents.

             http://wiki.apache.org/solr/UpdateXmlMessages

             maxDocs - Maximum number of documents to add since the last
                       commit before automatically triggering a new commit.

             maxTime - Maximum amount of time in ms that is allowed to pass
                       since a document was added before automatically
                       triggering a new commit.
             openSearcher - if false, the commit causes recent index changes
               to be flushed to stable storage, but does not cause a new
               searcher to be opened to make those changes visible.

             If the updateLog is enabled, then it's highly recommended to
             have some sort of hard autoCommit to limit the log size.
          -->
        <autoCommit>
            <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
            <openSearcher>false</openSearcher>
        </autoCommit>

        <!-- softAutoCommit is like autoCommit except it causes a
             'soft' commit which only ensures that changes are visible
             but does not ensure that data is synced to disk.  This is
             faster and more near-realtime friendly than a hard commit.
          -->
        <!-- setting softAutoCommit to 15 minutes for fft transcribedReports usage.          -->
        <autoSoftCommit>
            <maxTime>${solr.autoSoftCommit.maxTime:900000}</maxTime>
        </autoSoftCommit>

    </updateHandler>

    <!-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
         Query section - these settings control query time things like caches
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -->
    <query>
        <!-- Max Boolean Clauses

             Maximum number of clauses in each BooleanQuery,  an exception
             is thrown if exceeded.

             ** WARNING **

             This option actually modifies a global Lucene property that
             will affect all SolrCores.  If multiple solrconfig.xml files
             disagree on this property, the value at any given moment will
             be based on the last SolrCore to be initialized.

          -->
        <maxBooleanClauses>1024</maxBooleanClauses>


        <!-- Solr Internal Query Caches

             There are two implementations of cache available for Solr,
             LRUCache, based on a synchronized LinkedHashMap, and
             FastLRUCache, based on a ConcurrentHashMap.

             FastLRUCache has faster gets and slower puts in single
             threaded operation and thus is generally faster than LRUCache
             when the hit ratio of the cache is high (> 75%), and may be
             faster under other scenarios on multi-cpu systems.
        -->

        <!-- Filter Cache

             Cache used by SolrIndexSearcher for filters (DocSets),
             unordered sets of *all* documents that match a query.  When a
             new searcher is opened, its caches may be prepopulated or
             "autowarmed" using data from caches in the old searcher.
             autowarmCount is the number of items to prepopulate.  For
             LRUCache, the autowarmed items will be the most recently
             accessed items.

             Parameters:
               class - the SolrCache implementation LRUCache or
                   (LRUCache or FastLRUCache)
               size - the maximum number of entries in the cache
               initialSize - the initial capacity (number of entries) of
                   the cache.  (see java.util.HashMap)
               autowarmCount - the number of entries to prepopulate from
                   and old cache.
          -->
        <filterCache class="solr.FastLRUCache"
                     size="512"
                     initialSize="512"
                     autowarmCount="0"/>

        <!-- Query Result Cache

            Caches results of searches - ordered lists of document ids
            (DocList) based on a query, a sort, and the range of documents requested.
            Additional supported parameter by LRUCache:
               maxRamMB - the maximum amount of RAM (in MB) that this cache is allowed
                          to occupy
         -->
        <queryResultCache class="solr.LRUCache"
                          size="512"
                          initialSize="512"
                          autowarmCount="0"/>

        <!-- Document Cache

             Caches Lucene Document objects (the stored fields for each
             document).  Since Lucene internal document ids are transient,
             this cache will not be autowarmed.
          -->
        <documentCache class="solr.LRUCache"
                       size="512"
                       initialSize="512"
                       autowarmCount="0"/>

        <!-- custom cache currently used by block join -->
        <cache name="perSegFilter"
               class="solr.search.LRUCache"
               size="10"
               initialSize="0"
               autowarmCount="10"
               regenerator="solr.NoOpRegenerator"/>

        <!-- Lazy Field Loading

             If true, stored fields that are not requested will be loaded
             lazily.  This can result in a significant speed improvement
             if the usual case is to not load all stored fields,
             especially if the skipped fields are large compressed text
             fields.
        -->
        <enableLazyFieldLoading>true</enableLazyFieldLoading>

        <!-- Result Window Size

             An optimization for use with the queryResultCache.  When a search
             is requested, a superset of the requested number of document ids
             are collected.  For example, if a search for a particular query
             requests matching documents 10 through 19, and queryWindowSize is 50,
             then documents 0 through 49 will be collected and cached.  Any further
             requests in that range can be satisfied via the cache.
          -->
        <queryResultWindowSize>20</queryResultWindowSize>

        <!-- Maximum number of documents to cache for any entry in the
             queryResultCache.
          -->
        <queryResultMaxDocsCached>200</queryResultMaxDocsCached>

        <!-- Use Cold Searcher

             If a search request comes in and there is no current
             registered searcher, then immediately register the still
             warming searcher and use it.  If "false" then all requests
             will block until the first searcher is done warming.
          -->
        <useColdSearcher>false</useColdSearcher>

        <!-- Max Warming Searchers

             Maximum number of searchers that may be warming in the
             background concurrently.  An error is returned if this limit
             is exceeded.

             Recommend values of 1-2 for read-only slaves, higher for
             masters w/o cache warming.
          -->
        <maxWarmingSearchers>2</maxWarmingSearchers>

    </query>


    <!-- Request Dispatcher

         This section contains instructions for how the SolrDispatchFilter
         should behave when processing requests for this SolrCore.

         handleSelect is a legacy option that affects the behavior of requests
         such as /select?qt=XXX

         handleSelect="true" will cause the SolrDispatchFilter to process
         the request and dispatch the query to a handler specified by the
         "qt" param, assuming "/select" isn't already registered.

         handleSelect="false" will cause the SolrDispatchFilter to
         ignore "/select" requests, resulting in a 404 unless a handler
         is explicitly registered with the name "/select"

         handleSelect="true" is not recommended for new users, but is the default
         for backwards compatibility
      -->
    <requestDispatcher handleSelect="false">
        <!-- Request Parsing

             These settings indicate how Solr Requests may be parsed, and
             what restrictions may be placed on the ContentStreams from
             those requests

             enableRemoteStreaming - enables use of the stream.file
             and stream.url parameters for specifying remote streams.

             multipartUploadLimitInKB - specifies the max size (in KiB) of
             Multipart File Uploads that Solr will allow in a Request.

             formdataUploadLimitInKB - specifies the max size (in KiB) of
             form data (application/x-www-form-urlencoded) sent via
             POST. You can use POST to pass request parameters not
             fitting into the URL.

             addHttpRequestToContext - if set to true, it will instruct
             the requestParsers to include the original HttpServletRequest
             object in the context map of the SolrQueryRequest under the
             key "httpRequest". It will not be used by any of the existing
             Solr components, but may be useful when developing custom
             plugins.

             *** WARNING ***
             The settings below authorize Solr to fetch remote files, You
             should make sure your system has some authentication before
             using enableRemoteStreaming="true"

          -->
        <requestParsers enableRemoteStreaming="true"
                        multipartUploadLimitInKB="2048000"
                        formdataUploadLimitInKB="2048"
                        addHttpRequestToContext="false"/>

        <!-- HTTP Caching

             Set HTTP caching related parameters (for proxy caches and clients).

             The options below instruct Solr not to output any HTTP Caching
             related headers
          -->
        <httpCaching never304="true"/>

    </requestDispatcher>

    <!-- Request Handlers

         http://wiki.apache.org/solr/SolrRequestHandler

         Incoming queries will be dispatched to a specific handler by name
         based on the path specified in the request.

         Legacy behavior: If the request path uses "/select" but no Request
         Handler has that name, and if handleSelect="true" has been specified in
         the requestDispatcher, then the Request Handler is dispatched based on
         the qt parameter.  Handlers without a leading '/' are accessed this way
         like so: http://host/app/[core/]select?qt=name  If no qt is
         given, then the requestHandler that declares default="true" will be
         used or the one named "standard".

         If a Request Handler is declared with startup="lazy", then it will
         not be initialized until the first request that uses it.

      -->
    <!-- SearchHandler

         http://wiki.apache.org/solr/SearchHandler

         For processing Search Queries, the primary Request Handler
         provided with Solr is "SearchHandler" It delegates to a sequent
         of SearchComponents (see below) and supports distributed
         queries across multiple shards
      -->
    <requestHandler name="/select" class="solr.SearchHandler">
        <!-- default values for query parameters can be specified, these
             will be overridden by parameters in the request
          -->
        <lst name="defaults">
            <str name="echoParams">explicit</str>
            <int name="rows">10</int>
        </lst>

    </requestHandler>

    <!-- A request handler that returns indented JSON by default -->
    <requestHandler name="/query" class="solr.SearchHandler">
        <lst name="defaults">
            <str name="echoParams">explicit</str>
            <str name="wt">json</str>
            <str name="indent">true</str>
            <str name="df">text</str>
        </lst>
    </requestHandler>

    <initParams path="/update/**,/query,/select,/tvrh,/elevate,/spell">
        <lst name="defaults">
            <str name="df">text</str>
        </lst>
    </initParams>

    <!-- Field Analysis Request Handler

         RequestHandler that provides much the same functionality as
         analysis.jsp. Provides the ability to specify multiple field
         types and field names in the same request and outputs
         index-time and query-time analysis for each of them.

         Request parameters are:
         analysis.fieldname - field name whose analyzers are to be used

         analysis.fieldtype - field type whose analyzers are to be used
         analysis.fieldvalue - text for index-time analysis
         q (or analysis.q) - text for query time analysis
         analysis.showmatch (true|false) - When set to true and when
             query analysis is performed, the produced tokens of the
             field value analysis will be marked as "matched" for every
             token that is produces by the query analysis
     -->
    <requestHandler name="/analysis/field"
                    startup="lazy"
                    class="solr.FieldAnalysisRequestHandler"/>


    <!-- Document Analysis Handler

         http://wiki.apache.org/solr/AnalysisRequestHandler

         An analysis handler that provides a breakdown of the analysis
         process of provided documents. This handler expects a (single)
         content stream with the following format:

         <docs>
           <doc>
             <field name="id">1</field>
             <field name="name">The Name</field>
             <field name="text">The Text Value</field>
           </doc>
           <doc>...</doc>
           <doc>...</doc>
           ...
         </docs>

      Note: Each document must contain a field which serves as the
      unique key. This key is used in the returned response to associate
      an analysis breakdown to the analyzed document.

      Like the FieldAnalysisRequestHandler, this handler also supports
      query analysis by sending either an "analysis.query" or "q"
      request parameter that holds the query text to be analyzed. It
      also supports the "analysis.showmatch" parameter which when set to
      true, all field tokens that match the query tokens will be marked
      as a "match".
    -->
    <requestHandler name="/analysis/document"
                    class="solr.DocumentAnalysisRequestHandler"
                    startup="lazy"/>

    <!-- Echo the request contents back to the client -->
    <requestHandler name="/debug/dump" class="solr.DumpRequestHandler">
        <lst name="defaults">
            <str name="echoParams">explicit</str>
            <str name="echoHandler">true</str>
        </lst>
    </requestHandler>


    <!-- Search Components

         Search components are registered to SolrCore and used by
         instances of SearchHandler (which can access them by name)

         By default, the following components are available:

         <searchComponent name="query"     class="solr.QueryComponent" />
         <searchComponent name="facet"     class="solr.FacetComponent" />
         <searchComponent name="mlt"       class="solr.MoreLikeThisComponent" />
         <searchComponent name="highlight" class="solr.HighlightComponent" />
         <searchComponent name="stats"     class="solr.StatsComponent" />
         <searchComponent name="debug"     class="solr.DebugComponent" />

       -->
    <!-- Update Processors

       Chains of Update Processor Factories for dealing with Update
       Requests can be declared, and then used by name in Update
       Request Processors

       http://wiki.apache.org/solr/UpdateRequestProcessor

    -->

    <!-- This processing chain handles document expiry.
        The fft task that exports to solr specifies an expiryDate field
        and this configures solr to run a thread to delete
        documents that have passed their expiryDate
    -->
    <updateRequestProcessorChain name="FFTtranscribedReportsProcessorChain" default="true">
        <processor class="solr.IgnoreCommitOptimizeUpdateProcessorFactory">
          <int name="statusCode">200</int>
        </processor>
        <processor class="solr.processor.DocExpirationUpdateProcessorFactory">
            <int name="autoDeletePeriodSeconds">30</int>
            <str name="ttlFieldName">_ttl_</str>
            <str name="expirationFieldName">expiryDate</str>
        </processor>
        <processor class="solr.FirstFieldValueUpdateProcessorFactory">
            <str name="fieldName">expiryDate</str>
        </processor>
        <processor class="solr.LogUpdateProcessorFactory" />
        <processor class="solr.DistributedUpdateProcessorFactory" />
        <processor class="solr.RunUpdateProcessorFactory" />
    </updateRequestProcessorChain>

     <!-- Terms Component

         http://wiki.apache.org/solr/TermsComponent

         A component to return terms and document frequency of those
         terms
    -->
    <searchComponent name="terms" class="solr.TermsComponent"/>

    <!-- A request handler for demonstrating the terms component -->
    <requestHandler name="/terms" class="solr.SearchHandler" startup="lazy">
        <lst name="defaults">
            <bool name="terms">true</bool>
            <bool name="distrib">false</bool>
        </lst>
        <arr name="components">
            <str>terms</str>
        </arr>
    </requestHandler>

</config> 







-----Original Message-----
From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com] 
Sent: Friday, September 2, 2016 5:36 AM
To: solr-user@lucene.apache.org
Subject: Re: what to try next for replica that will not stay up.

My guess is that recovery is indeed successful but the leader is repeatedly marking this replicas as 'down' using what we call Leader-Initiated-Recovery or LIR. We need to understand why that is happening. Are there GC issues on this new node?

Can we see the logs on the leader from a bit before and after a line that should look like the following:
"Put replica core=documents_shard1_replica2 coreNodeName="...

Can we also see the complete cluster state?

On Thu, Sep 1, 2016 at 9:15 PM, Jon Hawkesworth < jon.hawkesworth@medquist.onmicrosoft.com> wrote:

> Hi
>
>
>
> If anyone has any suggestions of things I could try to resolve my 
> issue where one replica on one of my solcloud 6.0.1 shards refuses to 
> stay up, I'd love to hear them.  In fact, I'll get you something off 
> your amazon wishlist, within reason, if you can solve this puzzle.
>
>
>
> Today we pruned the dead replica, restarted the machine where it ran 
> and once the node had rejoined the cluster, we added a new replica.
>
> The replica was marked as Active for about 10 minutes then went down
>
>
>
> I put some example logging from below, but it looks much the same as 
> last time.
>
>
>
> There's a bunch of warnings about a checksum being different even 
> though the file size is the same and then RecoveryStrategy
>
> reports 'Could not publish as ACTIVE after succesful recovery'
>
>
>
> I think I've found where that message comes from in the code here:
> https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=
> blob;f=solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java;
> h= abd00aef19a731b42b314f8b526cdb2d77baf89f;hb=refs/heads/master
>
> (I am running 6.0.1 though so could have changed in latest devel).
>
>
>
> So it seems this chunk of codeā€¦
>
>
>
> 451         if (successfulRecovery) {
>
> 452           LOG.info("Registering as Active after recovery.");
>
> 453           try {
>
> 454             zkController.publish(core.getCoreDescriptor(),
> Replica.State.ACTIVE);
>
> 455           } catch (Exception e) {
>
> 456             LOG.error("Could not publish as ACTIVE after succesful
> recovery", e);
>
> 457             successfulRecovery = false;
>
> 458           }
>
> 459
>
>  460           if (successfulRecovery) {
>
> 461             close = true;
>
> 462             recoveryListener.recovered();
>
> 463           }
>
> 464         }
>
>
>
> results in this:
>
>
>
> org.apache.solr.common.SolrException: Cannot publish state of core 
> 'documents_shard1_replica2' as active without recovering first!
>
>                at org.apache.solr.cloud.ZkController.publish(
> ZkController.java:1141)
>
>                at org.apache.solr.cloud.ZkController.publish(
> ZkController.java:1097)
>
>                at org.apache.solr.cloud.ZkController.publish(
> ZkController.java:1093)
>
>                at org.apache.solr.cloud.RecoveryStrategy.doRecovery(
> RecoveryStrategy.java:457)
>
>                at org.apache.solr.cloud.RecoveryStrategy.run(
> RecoveryStrategy.java:224)
>
>                at 
> java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
>
>                at java.util.concurrent.FutureTask.run(Unknown Source)
>
>                at org.apache.solr.common.util.ExecutorUtil$
> MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>
>                at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
>
>                at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
>
>                at java.lang.Thread.run(Unknown Source)
>
>
>
> I don't yet understand the interaction with zookeeper but there's some 
> disagreement about whether recovery has happened or not (if it hadn't 
> from solr's point of view the successfulRecovery boolean would 
> presumably be false.
>
>
>
> Should I raise a JIRA?  Is there any other useful information I could 
> gather?
>
>
>
> I haven't really had any similar problems with the other 3 shards, 
> just shard1.
>
>
>
> The nodes that it is running on are all pretty similar - all vms built 
> to the same specification and the deployment of java and solrcloud is 
> automated so there shouldn't be any differences in the stack.
>
>
>
> Many thanks,
>
>
>
> Jon
>
>
>
>
>
>
>
>
>
> Example log output below
>
>
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnux.si did not match. expected checksum is 1186898951 and 
> actual is checksum 1994281621. expected length is 417 and actual 
> length is 417
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy.nvd did not match. expected checksum is 2200422612 and 
> actual is checksum 3635321041. expected length is 63 and actual length 
> is 65
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy.fdx did not match. expected checksum is 281622189 and 
> actual is checksum 838341528. expected length is 84 and actual length 
> is 84
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy.nvm did not match. expected checksum is 1875012021 and 
> actual is checksum 524812847. expected length is 108 and actual length 
> is 108
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy.fnm did not match. expected checksum is 1681449973 and 
> actual is checksum 3351426142. expected length is 1265 and actual 
> length is 1265
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy_Lucene54_0.dvm did not match. expected checksum is 
> 355987228 and actual is checksum 847034886. expected length is 380 and 
> actual length is 404
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy_Lucene50_0.pos did not match. expected checksum is 
> 806636274 and actual is checksum 2272195325. expected length is 1059 
> and actual length is 1172
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy_Lucene50_0.doc did not match. expected checksum is 
> 4041316671 and actual is checksum 3122885740. expected length is 212 
> and actual length is 281
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy_Lucene50_0.tim did not match. expected checksum is 
> 2891628412 and actual is checksum 2420913910. expected length is 5346 
> and actual length is 6251
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy_Lucene50_0.tip did not match. expected checksum is 
> 1652105503 and actual is checksum 807238796. expected length is 336 
> and actual length is 349
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy_Lucene54_0.dvd did not match. expected checksum is 
> 2664049801 and actual is checksum 2930561414. expected length is 130 
> and actual length is 145
>
> 9/1/2016, 12:37:06 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy.fdt did not match. expected checksum is 4175958592 and 
> actual is checksum 3650490510. expected length is 4280 and actual 
> length is 4983
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuy.si did not match. expected checksum is 2223401636 and 
> actual is checksum 734463570. expected length is 535 and actual length 
> is 535
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz_Lucene54_0.dvd did not match. expected checksum is 
> 202072236 and actual is checksum 4194802930. expected length is 96 and 
> actual length is 264
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz_Lucene50_0.tip did not match. expected checksum is 
> 2123658306 and actual is checksum 435878007. expected length is 298 
> and actual length is 639
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz.nvd did not match. expected checksum is 4214748910 and 
> actual is checksum 3784036105. expected length is 59 and actual length 
> is 77
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz.fdt did not match. expected checksum is 3837568601 and 
> actual is checksum 2542454689. expected length is 896 and actual 
> length is 20338
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz.fdx did not match. expected checksum is 2070429440 and 
> actual is checksum 3279752998. expected length is 84 and actual length 
> is 86
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz_Lucene50_0.pos did not match. expected checksum is 
> 2299588010 and actual is checksum 2299553846. expected length is 190 
> and actual length is 5717
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz_Lucene54_0.dvm did not match. expected checksum is 
> 914650440 and actual is checksum 2852383192. expected length is 312 
> and actual length is 548
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz.nvm did not match. expected checksum is 3037735995 and 
> actual is checksum 4023026424. expected length is 108 and actual 
> length is 108
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz_Lucene50_0.doc did not match. expected checksum is 
> 3813274592 and actual is checksum 189237707. expected length is 110 
> and actual length is 1945
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz_Lucene50_0.tim did not match. expected checksum is 
> 3013245878 and actual is checksum 2122722316. expected length is 1757 
> and actual length is 16642
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz.fnm did not match. expected checksum is 2117653105 and 
> actual is checksum 3401755804. expected length is 1265 and actual 
> length is 1265
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnuz.si did not match. expected checksum is 2715978927 and 
> actual is checksum 3653125964. expected length is 535 and actual 
> length is 535
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0.fdt did not match. expected checksum is 1699019853 and 
> actual is checksum 3731775500. expected length is 15865 and actual 
> length is 12728
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0_Lucene50_0.pos did not match. expected checksum is 
> 2189908204 and actual is checksum 2338139479. expected length is 4475 
> and actual length is 3431
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0_Lucene50_0.doc did not match. expected checksum is 
> 1522019614 and actual is checksum 969681917. expected length is 1394 
> and actual length is 1093
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0_Lucene50_0.tim did not match. expected checksum is 
> 813529901 and actual is checksum 529669468. expected length is 13843 
> and actual length is 12535
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0.si did not match. expected checksum is 3802482417 and 
> actual is checksum 1865633126. expected length is 535 and actual 
> length is 535
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0_Lucene54_0.dvm did not match. expected checksum is 
> 4236057860 and actual is checksum 2986112802. expected length is 500 
> and actual length is 476
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0.fdx did not match. expected checksum is 2497099401 and 
> actual is checksum 990046808. expected length is 85 and actual length 
> is 84
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0.nvd did not match. expected checksum is 1736308969 and 
> actual is checksum 3657480551. expected length is 73 and actual length 
> is 71
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0_Lucene50_0.tip did not match. expected checksum is 
> 1362235492 and actual is checksum 640196019. expected length is 570 
> and actual length is 531
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0.fnm did not match. expected checksum is 1975043794 and 
> actual is checksum 1035049893. expected length is 1265 and actual 
> length is 1265
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0.nvm did not match. expected checksum is 2985228383 and 
> actual is checksum 2603407196. expected length is 108 and actual 
> length is 108
>
> 9/1/2016, 12:37:07 PM
>
>
>
> WARN false
>
>
>
>
>
> IndexFetcher
>
>
>
> File _jnv0_Lucene54_0.dvd did not match. expected checksum is 
> 762056409 and actual is checksum 1514176651. expected length is 228 
> and actual length is 211
>
> 9/1/2016, 12:37:09 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Starting log replay tlog{file=E:\solr_home\transcribedReports_shard1_
> replica2\data\tlog\tlog.0000000000000000005 refcount=2} active=true 
> starting pos=20222
>
> 9/1/2016, 12:37:10 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Log replay finished. recoveryInfo=RecoveryInfo{adds=10 deletes=0
> deleteByQuery=1 errors=0 positionOfStart=20222}
>
> 9/1/2016, 12:37:10 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Could not publish as ACTIVE after succesful recovery
>
> 9/1/2016, 12:37:10 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Recovery failed - trying again... (0)
>
> 9/1/2016, 12:37:37 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Starting log replay tlog{file=E:\solr_home\transcribedReports_shard1_
> replica2\data\tlog\tlog.0000000000000000006 refcount=2} active=true 
> starting pos=0
>
> 9/1/2016, 12:37:38 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Log replay finished. recoveryInfo=RecoveryInfo{adds=2 deletes=0
> deleteByQuery=0 errors=0 positionOfStart=0}
>
> 9/1/2016, 12:37:41 PM
>
>
>
> WARN false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Stopping recovery for core=[transcribedReports_shard1_replica2]
> coreNodeName=[core_node14]
>
> 9/1/2016, 12:42:13 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Starting log replay tlog{file=E:\solr_home\transcribedReports_shard1_
> replica2\data\tlog\tlog.0000000000000000007 refcount=2} active=true 
> starting pos=1748
>
> 9/1/2016, 12:42:14 PM
>
>
>
> WARN false
>
>
>
>
>
> UpdateLog
>
>
>
> Log replay finished. recoveryInfo=RecoveryInfo{adds=12 deletes=0
> deleteByQuery=0 errors=0 positionOfStart=1748}
>
> 9/1/2016, 12:42:14 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Could not publish as ACTIVE after succesful recovery
>
> 9/1/2016, 12:42:14 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Recovery failed - trying again... (0)
>
> 9/1/2016, 12:42:43 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Could not publish as ACTIVE after succesful recovery
>
> 9/1/2016, 12:42:43 PM
>
>
>
> ERROR false
>
>
>
>
>
> RecoveryStrategy
>
>
>
> Recovery failed - trying again... (0)
>
>
>
> *Jon Hawkesworth*
> Software Developer
>
>
>
>
>
> Hanley Road, Malvern, WR13 6NP. UK
>
> O: +44 (0) 1684 312313
>
> *jon.hawkesworth@mmodal.com <jon.hawkesworth@mmodal.com> 
> www.mmodal.com
> <http://www.medquist.com/>*
>
>
>
> *This electronic mail transmission contains confidential information 
> intended only for the person(s) named. Any use, distribution, copying 
> or disclosure by another person is strictly prohibited. If you are not 
> the intended recipient of this e-mail, promptly delete it and all 
> attachments.*
>
>
>



--
Regards,
Shalin Shekhar Mangar.
Mime
View raw message