lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0-ea-b79) - Build # 4702 - Still Failing!
Date Sat, 16 Mar 2013 17:16:25 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/4702/
Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseParallelGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=200, name=coreLoadExecutor-88-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=200, name=coreLoadExecutor-88-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1569)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:34)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:272)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:224)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:744)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
	at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
	at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
	at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@21b2fe58 lockFactory=org.apache.lucene.store.NativeFSLockFactory@f110421)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=202, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=202, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1569)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:878)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:870)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:34)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:272)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:224)
	at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
	at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.RAMDirectory@34a32a11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19085b9d)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)


REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
Server at http://127.0.0.1:35254/lrooy/onenodecollectioncore returned non ok status:404, message:Can not find: /lrooy/onenodecollectioncore/update

Stack Trace:
org.apache.solr.common.SolrException: Server at http://127.0.0.1:35254/lrooy/onenodecollectioncore returned non ok status:404, message:Can not find: /lrooy/onenodecollectioncore/update
	at __randomizedtesting.SeedInfo.seed([DDA8E93F6FE5B250:5C4E672718BAD26C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:375)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:89)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:487)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)


REGRESSION:  org.apache.solr.cloud.UnloadDistributedZkTest.testDistribSearch

Error Message:
Still found shard

Stack Trace:
java.lang.AssertionError: Still found shard
	at __randomizedtesting.SeedInfo.seed([DDA8E93F6FE5B250:5C4E672718BAD26C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:124)
	at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:77)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:487)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 8933 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4]   2> 0 T141 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 4 T141 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1363452959107
[junit4:junit4]   2> 5 T141 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 6 T142 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 7 T142 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 7 T142 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 8 T142 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 8 T142 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 9 T142 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1363452959107/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 106 T141 oasc.ZkTestServer.run start zk server on port:48954
[junit4:junit4]   2> 107 T141 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48954 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6bc7026d
[junit4:junit4]   2> 108 T141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 108 T147 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48954. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 109 T147 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48954, initiating session
[junit4:junit4]   2> 109 T143 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57277
[junit4:junit4]   2> 110 T143 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57277
[junit4:junit4]   2> 110 T145 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 208 T145 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d742119880000 with negotiated timeout 10000 for client /127.0.0.1:57277
[junit4:junit4]   2> 208 T147 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48954, sessionid = 0x13d742119880000, negotiated timeout = 10000
[junit4:junit4]   2> 209 T148 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bc7026d name:ZooKeeperConnection Watcher:127.0.0.1:48954 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 210 T141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 210 T141 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 214 T146 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d742119880000
[junit4:junit4]   2> 215 T141 oaz.ZooKeeper.close Session: 0x13d742119880000 closed
[junit4:junit4]   2> 215 T143 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57277 which had sessionid 0x13d742119880000
[junit4:junit4]   2> 216 T141 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48954/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6500f165
[junit4:junit4]   2> 215 T148 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 216 T141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 217 T149 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48954. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 217 T149 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48954, initiating session
[junit4:junit4]   2> 217 T143 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57278
[junit4:junit4]   2> 218 T143 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57278
[junit4:junit4]   2> 219 T145 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d742119880001 with negotiated timeout 10000 for client /127.0.0.1:57278
[junit4:junit4]   2> 219 T149 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48954, sessionid = 0x13d742119880001, negotiated timeout = 10000
[junit4:junit4]   2> 219 T150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6500f165 name:ZooKeeperConnection Watcher:127.0.0.1:48954/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 220 T141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 220 T141 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 224 T141 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 227 T141 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 230 T141 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 233 T141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 234 T141 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 239 T141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 242 T141 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 247 T141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 247 T141 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 251 T141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 252 T141 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 255 T141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 256 T141 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 259 T141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 259 T141 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 262 T141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 263 T141 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 267 T141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 267 T141 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 270 T141 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 271 T141 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 274 T146 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d742119880001
[junit4:junit4]   2> 274 T150 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 274 T141 oaz.ZooKeeper.close Session: 0x13d742119880001 closed
[junit4:junit4]   2> 274 T143 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57278 which had sessionid 0x13d742119880001
[junit4:junit4]   2> 336 T141 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 343 T141 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45612
[junit4:junit4]   2> 344 T141 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 344 T141 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 345 T141 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363452959377
[junit4:junit4]   2> 345 T141 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363452959377/solr.xml
[junit4:junit4]   2> 345 T141 oasc.CoreContainer.<init> New CoreContainer 1038946732
[junit4:junit4]   2> 346 T141 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363452959377/'
[junit4:junit4]   2> 346 T141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363452959377/'
[junit4:junit4]   2> 367 T141 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 367 T141 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 368 T141 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 368 T141 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 368 T141 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 369 T141 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 369 T141 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 369 T141 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 370 T141 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 370 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 379 T141 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 391 T141 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:48954/solr
[junit4:junit4]   2> 391 T141 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 392 T141 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48954 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@6b9def64
[junit4:junit4]   2> 392 T141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 393 T160 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48954. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 393 T160 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48954, initiating session
[junit4:junit4]   2> 393 T143 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57279
[junit4:junit4]   2> 394 T143 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57279
[junit4:junit4]   2> 395 T145 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d742119880002 with negotiated timeout 20000 for client /127.0.0.1:57279
[junit4:junit4]   2> 395 T160 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48954, sessionid = 0x13d742119880002, negotiated timeout = 20000
[junit4:junit4]   2> 395 T161 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b9def64 name:ZooKeeperConnection Watcher:127.0.0.1:48954 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 395 T141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 397 T146 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d742119880002
[junit4:junit4]   2> 398 T161 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 398 T143 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57279 which had sessionid 0x13d742119880002
[junit4:junit4]   2> 398 T141 oaz.ZooKeeper.close Session: 0x13d742119880002 closed
[junit4:junit4]   2> 399 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 402 T141 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48954/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@2f54e214
[junit4:junit4]   2> 403 T141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 403 T162 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48954. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 404 T162 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48954, initiating session
[junit4:junit4]   2> 404 T143 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57280
[junit4:junit4]   2> 404 T143 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57280
[junit4:junit4]   2> 405 T145 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d742119880003 with negotiated timeout 20000 for client /127.0.0.1:57280
[junit4:junit4]   2> 405 T162 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48954, sessionid = 0x13d742119880003, negotiated timeout = 20000
[junit4:junit4]   2> 405 T163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f54e214 name:ZooKeeperConnection Watcher:127.0.0.1:48954/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 406 T141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 407 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 409 T141 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 411 T141 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45612_
[junit4:junit4]   2> 412 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:45612_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:45612_
[junit4:junit4]   2> 412 T141 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45612_
[junit4:junit4]   2> 415 T141 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 421 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 422 T141 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 424 T141 oasc.Overseer.start Overseer (id=89355253128364035-127.0.0.1:45612_-n_0000000000) starting
[junit4:junit4]   2> 425 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 426 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 426 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 427 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 428 T165 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 429 T141 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 432 T141 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 434 T141 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 437 T164 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 443 T166 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363452959377/collection1
[junit4:junit4]   2> 443 T166 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 444 T166 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 445 T166 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 446 T166 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363452959377/collection1/'
[junit4:junit4]   2> 447 T166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363452959377/collection1/lib/README' to classloader
[junit4:junit4]   2> 447 T166 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363452959377/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 482 T166 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 523 T166 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 525 T166 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 532 T166 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 901 T166 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 909 T166 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 911 T166 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 934 T166 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 938 T166 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 941 T166 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 943 T166 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 943 T166 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 944 T166 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 945 T166 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 945 T166 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 946 T166 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 946 T166 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363452959377/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/control/data/
[junit4:junit4]   2> 946 T166 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4de60052
[junit4:junit4]   2> 947 T166 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 948 T166 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363452959107/control/data forceNew: false
[junit4:junit4]   2> 948 T166 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/control/data/index/
[junit4:junit4]   2> 948 T166 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363452959107/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 948 T166 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363452959107/control/data/index forceNew: false
[junit4:junit4]   2> 950 T166 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3544ebd9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a2065d1),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 950 T166 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 952 T166 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 953 T166 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 953 T166 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 954 T166 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 954 T166 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 954 T166 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 955 T166 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 955 T166 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 955 T166 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 958 T166 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 960 T166 oass.SolrIndexSearcher.<init> Opening Searcher@1169e8c7 main
[junit4:junit4]   2> 961 T166 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/control/data/tlog
[junit4:junit4]   2> 961 T166 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 962 T166 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 965 T167 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1169e8c7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 967 T166 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 967 T166 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 968 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 1943 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1944 T164 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:45612",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:45612_",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core":"collection1"}
[junit4:junit4]   2> 1945 T164 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 1954 T164 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 1955 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 1962 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 1971 T166 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 1972 T166 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45612 collection:control_collection shard:shard1
[junit4:junit4]   2> 1976 T166 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 1984 T166 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 1991 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 1992 T166 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 1993 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1995 T166 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 1996 T166 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 1997 T166 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45612/collection1/
[junit4:junit4]   2> 1998 T166 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 1998 T166 oasc.SyncStrategy.syncToMe http://127.0.0.1:45612/collection1/ has no replicas
[junit4:junit4]   2> 1998 T166 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45612/collection1/
[junit4:junit4]   2> 1999 T166 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2004 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3467 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3479 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3511 T166 oasc.ZkController.register We are http://127.0.0.1:45612/collection1/ and leader is http://127.0.0.1:45612/collection1/
[junit4:junit4]   2> 3512 T166 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45612
[junit4:junit4]   2> 3512 T166 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3512 T166 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3513 T166 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3515 T166 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3517 T141 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3517 T141 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3518 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3522 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3523 T141 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48954/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@42b1a2ce
[junit4:junit4]   2> 3524 T141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3524 T169 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48954. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 3525 T169 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48954, initiating session
[junit4:junit4]   2> 3525 T143 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57282
[junit4:junit4]   2> 3526 T143 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57282
[junit4:junit4]   2> 3527 T145 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d742119880004 with negotiated timeout 10000 for client /127.0.0.1:57282
[junit4:junit4]   2> 3527 T169 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48954, sessionid = 0x13d742119880004, negotiated timeout = 10000
[junit4:junit4]   2> 3527 T170 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42b1a2ce name:ZooKeeperConnection Watcher:127.0.0.1:48954/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3528 T141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3530 T141 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3534 T141 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 3594 T141 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 3596 T141 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42845
[junit4:junit4]   2> 3597 T141 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3598 T141 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3598 T141 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363452962636
[junit4:junit4]   2> 3598 T141 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363452962636/solr.xml
[junit4:junit4]   2> 3599 T141 oasc.CoreContainer.<init> New CoreContainer 339749098
[junit4:junit4]   2> 3599 T141 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363452962636/'
[junit4:junit4]   2> 3599 T141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363452962636/'
[junit4:junit4]   2> 3618 T141 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 3618 T141 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3619 T141 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 3619 T141 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3619 T141 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3620 T141 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3620 T141 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3620 T141 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3621 T141 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3621 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 3629 T141 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3647 T141 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:48954/solr
[junit4:junit4]   2> 3648 T141 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 3648 T141 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48954 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@10343c23
[junit4:junit4]   2> 3649 T141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3649 T180 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48954. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 3650 T180 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48954, initiating session
[junit4:junit4]   2> 3650 T143 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57283
[junit4:junit4]   2> 3651 T143 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57283
[junit4:junit4]   2> 3652 T145 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d742119880005 with negotiated timeout 20000 for client /127.0.0.1:57283
[junit4:junit4]   2> 3652 T180 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48954, sessionid = 0x13d742119880005, negotiated timeout = 20000
[junit4:junit4]   2> 3652 T181 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10343c23 name:ZooKeeperConnection Watcher:127.0.0.1:48954 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3653 T141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3654 T146 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d742119880005
[junit4:junit4]   2> 3655 T181 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 3655 T143 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57283 which had sessionid 0x13d742119880005
[junit4:junit4]   2> 3655 T141 oaz.ZooKeeper.close Session: 0x13d742119880005 closed
[junit4:junit4]   2> 3657 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 3661 T141 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48954/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@4065286
[junit4:junit4]   2> 3662 T141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3662 T182 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48954. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 3663 T182 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48954, initiating session
[junit4:junit4]   2> 3663 T143 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57284
[junit4:junit4]   2> 3663 T143 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57284
[junit4:junit4]   2> 3664 T145 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d742119880006 with negotiated timeout 20000 for client /127.0.0.1:57284
[junit4:junit4]   2> 3665 T182 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48954, sessionid = 0x13d742119880006, negotiated timeout = 20000
[junit4:junit4]   2> 3665 T183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4065286 name:ZooKeeperConnection Watcher:127.0.0.1:48954/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3665 T141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3667 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3668 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3670 T141 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 4674 T141 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42845_
[junit4:junit4]   2> 4675 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:42845_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:42845_
[junit4:junit4]   2> 4675 T141 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42845_
[junit4:junit4]   2> 4679 T183 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4679 T170 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4679 T163 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4679 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 4685 T184 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363452962636/collection1
[junit4:junit4]   2> 4686 T184 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 4686 T184 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 4687 T184 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 4688 T184 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363452962636/collection1/'
[junit4:junit4]   2> 4689 T184 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363452962636/collection1/lib/README' to classloader
[junit4:junit4]   2> 4689 T184 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363452962636/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 4715 T184 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 4748 T184 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 4749 T184 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4755 T184 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4986 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4988 T164 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "core_node_name":"1",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:45612",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:45612_",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core":"collection1"}
[junit4:junit4]   2> 4996 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 4996 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 4996 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5119 T184 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5125 T184 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5127 T184 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5150 T184 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5153 T184 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5157 T184 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5158 T184 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5159 T184 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5159 T184 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5160 T184 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5161 T184 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5161 T184 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5161 T184 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363452962636/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty1/
[junit4:junit4]   2> 5161 T184 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4de60052
[junit4:junit4]   2> 5162 T184 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5163 T184 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty1 forceNew: false
[junit4:junit4]   2> 5163 T184 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty1/index/
[junit4:junit4]   2> 5163 T184 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5164 T184 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty1/index forceNew: false
[junit4:junit4]   2> 5165 T184 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5640b971 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3e5045db),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5166 T184 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 5168 T184 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5168 T184 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5169 T184 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5169 T184 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5170 T184 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5170 T184 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5171 T184 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5171 T184 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5172 T184 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5174 T184 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5176 T184 oass.SolrIndexSearcher.<init> Opening Searcher@85fbdf4 main
[junit4:junit4]   2> 5177 T184 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty1/tlog
[junit4:junit4]   2> 5177 T184 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5178 T184 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5186 T185 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@85fbdf4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5186 T184 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 5187 T184 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6504 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6505 T164 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:42845",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:42845_",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core":"collection1"}
[junit4:junit4]   2> 6505 T164 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 6505 T164 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6580 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6580 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6580 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7189 T184 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7190 T184 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42845 collection:collection1 shard:shard1
[junit4:junit4]   2> 7191 T184 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7205 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7206 T184 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7206 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7239 T184 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7240 T184 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7240 T184 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42845/collection1/
[junit4:junit4]   2> 7241 T184 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7241 T184 oasc.SyncStrategy.syncToMe http://127.0.0.1:42845/collection1/ has no replicas
[junit4:junit4]   2> 7242 T184 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42845/collection1/
[junit4:junit4]   2> 7242 T184 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7250 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8085 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8097 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8097 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8098 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8121 T184 oasc.ZkController.register We are http://127.0.0.1:42845/collection1/ and leader is http://127.0.0.1:42845/collection1/
[junit4:junit4]   2> 8121 T184 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42845
[junit4:junit4]   2> 8122 T184 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8122 T184 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8122 T184 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8124 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8126 T141 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 8127 T141 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8127 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8203 T141 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8205 T141 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58139
[junit4:junit4]   2> 8205 T141 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8206 T141 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8206 T141 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237
[junit4:junit4]   2> 8206 T141 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/solr.xml
[junit4:junit4]   2> 8207 T141 oasc.CoreContainer.<init> New CoreContainer 1463596936
[junit4:junit4]   2> 8207 T141 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/'
[junit4:junit4]   2> 8208 T141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/'
[junit4:junit4]   2> 8225 T141 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8225 T141 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8226 T141 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8226 T141 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8226 T141 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8227 T141 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8227 T141 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8227 T141 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8228 T141 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8228 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8234 T141 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8243 T141 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:48954/solr
[junit4:junit4]   2> 8244 T141 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8244 T141 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48954 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@b11b129
[junit4:junit4]   2> 8245 T141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8245 T196 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48954. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 8246 T196 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48954, initiating session
[junit4:junit4]   2> 8246 T143 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57285
[junit4:junit4]   2> 8246 T143 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57285
[junit4:junit4]   2> 8248 T145 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d742119880007 with negotiated timeout 20000 for client /127.0.0.1:57285
[junit4:junit4]   2> 8248 T196 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48954, sessionid = 0x13d742119880007, negotiated timeout = 20000
[junit4:junit4]   2> 8249 T197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b11b129 name:ZooKeeperConnection Watcher:127.0.0.1:48954 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8249 T141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8251 T146 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d742119880007
[junit4:junit4]   2> 8258 T197 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8258 T143 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57285 which had sessionid 0x13d742119880007
[junit4:junit4]   2> 8258 T141 oaz.ZooKeeper.close Session: 0x13d742119880007 closed
[junit4:junit4]   2> 8259 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8264 T141 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48954/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@4e9fafda
[junit4:junit4]   2> 8265 T141 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8265 T198 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:48954. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 8266 T198 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48954, initiating session
[junit4:junit4]   2> 8266 T143 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:57286
[junit4:junit4]   2> 8266 T143 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:57286
[junit4:junit4]   2> 8268 T145 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d742119880008 with negotiated timeout 20000 for client /127.0.0.1:57286
[junit4:junit4]   2> 8268 T198 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:48954, sessionid = 0x13d742119880008, negotiated timeout = 20000
[junit4:junit4]   2> 8269 T199 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e9fafda name:ZooKeeperConnection Watcher:127.0.0.1:48954/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8269 T141 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8271 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8279 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8283 T141 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9288 T141 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58139_
[junit4:junit4]   2> 9289 T146 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d742119880008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:58139_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:58139_
[junit4:junit4]   2> 9299 T141 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58139_
[junit4:junit4]   2> 9302 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9303 T170 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9303 T199 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9303 T183 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9304 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9304 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9305 T163 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9312 T200 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/collection1
[junit4:junit4]   2> 9312 T200 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9313 T200 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9313 T200 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9315 T200 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/collection1/'
[junit4:junit4]   2> 9315 T200 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/collection1/lib/README' to classloader
[junit4:junit4]   2> 9315 T200 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9341 T200 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 9377 T200 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 9378 T200 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 9384 T200 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 9606 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9607 T164 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "core_node_name":"2",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:42845",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:42845_",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core":"collection1"}
[junit4:junit4]   2> 9612 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9612 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9612 T199 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9612 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9721 T200 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 9731 T200 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 9734 T200 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 9761 T200 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9766 T200 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9770 T200 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9772 T200 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 9772 T200 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 9773 T200 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9774 T200 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 9774 T200 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 9775 T200 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 9775 T200 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/
[junit4:junit4]   2> 9775 T200 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4de60052
[junit4:junit4]   2> 9776 T200 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 9777 T200 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2 forceNew: false
[junit4:junit4]   2> 9777 T200 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/index/
[junit4:junit4]   2> 9777 T200 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 9778 T200 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/index forceNew: false
[junit4:junit4]   2> 9780 T200 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@27c5c202 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c40d827),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9781 T200 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 9783 T200 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 9784 T200 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 9785 T200 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 9786 T200 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 9786 T200 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 9787 T200 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 9787 T200 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 9788 T200 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 9788 T200 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 9792 T200 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 9795 T200 oass.SolrIndexSearcher.<init> Opening Searcher@64de09f1 main
[junit4:junit4]   2> 9796 T200 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/tlog
[junit4:junit4]   2> 9797 T200 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 9797 T200 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 9801 T201 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@64de09f1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 9804 T200 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 9804 T200 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11124 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11126 T164 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:58139",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:58139_",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core":"collection1"}
[junit4:junit4]   2> 11126 T164 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 11126 T164 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 11157 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11157 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11158 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11157 T199 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11808 T200 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11809 T200 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58139 collection:collection1 shard:shard1
[junit4:junit4]   2> 11816 T200 oasc.ZkController.register We are http://127.0.0.1:58139/collection1/ and leader is http://127.0.0.1:42845/collection1/
[junit4:junit4]   2> 11817 T200 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58139
[junit4:junit4]   2> 11817 T200 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 11818 T200 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@3b84b706 url=http://127.0.0.1:58139/collection1 node=127.0.0.1:58139_ C7_STATE=coll:collection1 core:collection1 props:{state=down, collection=collection1, base_url=http://127.0.0.1:58139, node_name=127.0.0.1:58139_, core=collection1}
[junit4:junit4]   2> 11824 T202 C7 P58139 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 11824 T200 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11825 T202 C7 P58139 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 11826 T202 C7 P58139 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 11827 T202 C7 P58139 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11827 T141 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 11828 T141 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 11829 T202 C7 P58139 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 11829 T141 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 11838 T141 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C8 name=collection1 org.apache.solr.core.SolrCore@4ed2d38e url=http://127.0.0.1:45612/collection1 node=127.0.0.1:45612_ C8_STATE=coll:control_collection core:collection1 props:{state=active, collection=control_collection, base_url=http://127.0.0.1:45612, shard=shard1, node_name=127.0.0.1:45612_, core=collection1, leader=true}
[junit4:junit4]   2> 11868 T158 C8 P45612 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3544ebd9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a2065d1),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 11870 T158 C8 P45612 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 11878 T159 C8 P45612 /update {version=2&wt=javabin} {add=[1 (1429684062485741568)]} 0 22
[junit4:junit4]   2> 11881 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10001 (1429684062485741569)]} 0 25
[junit4:junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@546fcbb4 url=http://127.0.0.1:42845/collection1 node=127.0.0.1:42845_ C9_STATE=coll:collection1 core:collection1 props:{state=active, collection=collection1, base_url=http://127.0.0.1:42845, shard=shard1, node_name=127.0.0.1:42845_, core=collection1, leader=true}
[junit4:junit4]   2> 11901 T177 C9 P42845 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5640b971 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3e5045db),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 11903 T177 C9 P42845 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 11908 T178 C9 P42845 /update {version=2&wt=javabin} {add=[1 (1429684062520344577)]} 0 10
[junit4:junit4]   2> 11911 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10001 (1429684062520344576)]} 0 13
[junit4:junit4]   2> 11954 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[1 (-1429684062578016256)]} 0 1
[junit4:junit4]   2> 11956 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[10001 (-1429684062579064832)]} 0 1
[junit4:junit4]   2> 11960 T176 C9 P42845 /update {version=2&wt=javabin} {delete=[1 (-1429684062584307712)]} 0 1
[junit4:junit4]   2> 11961 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10001 (-1429684062585356288)]} 0 1
[junit4:junit4]   2> 11969 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10002 (1429684062590599168)]} 0 4
[junit4:junit4]   2> 11970 T157 C8 P45612 /update {version=2&wt=javabin} {add=[2 (1429684062590599169)]} 0 5
[junit4:junit4]   2> 11974 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10002 (1429684062596890624)]} 0 3
[junit4:junit4]   2> 11977 T176 C9 P42845 /update {version=2&wt=javabin} {add=[2 (1429684062597939200)]} 0 4
[junit4:junit4]   2> 11996 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10003 (1429684062617862144)]} 0 4
[junit4:junit4]   2> 11999 T159 C8 P45612 /update {version=2&wt=javabin} {add=[3 (1429684062622056448)]} 0 4
[junit4:junit4]   2> 12003 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10003 (1429684062626250752)]} 0 3
[junit4:junit4]   2> 12006 T177 C9 P42845 /update {version=2&wt=javabin} {add=[3 (1429684062629396480)]} 0 3
[junit4:junit4]   2> 12021 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10004 (1429684062644076544)]} 0 5
[junit4:junit4]   2> 12023 T157 C8 P45612 /update {version=2&wt=javabin} {add=[4 (1429684062647222272)]} 0 4
[junit4:junit4]   2> 12030 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10004 (1429684062653513728)]} 0 5
[junit4:junit4]   2> 12032 T177 C9 P42845 /update {version=2&wt=javabin} {add=[4 (1429684062655610880)]} 0 5
[junit4:junit4]   2> 12096 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10005 (1429684062724816896)]} 0 2
[junit4:junit4]   2> 12098 T159 C8 P45612 /update {version=2&wt=javabin} {add=[5 (1429684062726914048)]} 0 3
[junit4:junit4]   2> 12100 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10005 (1429684062730059776)]} 0 1
[junit4:junit4]   2> 12102 T178 C9 P42845 /update {version=2&wt=javabin} {add=[5 (1429684062732156928)]} 0 2
[junit4:junit4]   2> 12120 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10006 (1429684062749982720)]} 0 2
[junit4:junit4]   2> 12121 T157 C8 P45612 /update {version=2&wt=javabin} {add=[6 (1429684062751031296)]} 0 2
[junit4:junit4]   2> 12125 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10006 (1429684062755225600)]} 0 2
[junit4:junit4]   2> 12126 T176 C9 P42845 /update {version=2&wt=javabin} {add=[6 (1429684062756274176)]} 0 2
[junit4:junit4]   2> 12198 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[10003 (-1429684062832820224)]} 0 1
[junit4:junit4]   2> 12198 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[3 (-1429684062833868800)]} 0 0
[junit4:junit4]   2> 12203 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10003 (-1429684062838063104)]} 0 1
[junit4:junit4]   2> 12203 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[3 (-1429684062839111680)]} 0 1
[junit4:junit4]   2> 12210 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10007 (1429684062843305984)]} 0 3
[junit4:junit4]   2> 12212 T157 C8 P45612 /update {version=2&wt=javabin} {add=[7 (1429684062844354560)]} 0 5
[junit4:junit4]   2> 12218 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10007 (1429684062851694592)]} 0 4
[junit4:junit4]   2> 12220 T178 C9 P42845 /update {version=2&wt=javabin} {add=[7 (1429684062853791744)]} 0 4
[junit4:junit4]   2> 12229 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[10004 (-1429684062866374656)]} 0 1
[junit4:junit4]   2> 12232 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[4 (-1429684062869520384)]} 0 1
[junit4:junit4]   2> 12234 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[10004 (-1429684062871617536)]} 0 1
[junit4:junit4]   2> 12238 T176 C9 P42845 /update {version=2&wt=javabin} {delete=[4 (-1429684062874763264)]} 0 2
[junit4:junit4]   2> 12241 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10008 (1429684062876860416)]} 0 3
[junit4:junit4]   2> 12245 T157 C8 P45612 /update {version=2&wt=javabin} {add=[8 (1429684062880006144)]} 0 3
[junit4:junit4]   2> 12249 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10008 (1429684062884200448)]} 0 4
[junit4:junit4]   2> 12253 T177 C9 P42845 /update {version=2&wt=javabin} {add=[8 (1429684062888394752)]} 0 3
[junit4:junit4]   2> 12323 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10009 (1429684062962843648)]} 0 2
[junit4:junit4]   2> 12327 T159 C8 P45612 /update {version=2&wt=javabin} {add=[9 (1429684062967037952)]} 0 3
[junit4:junit4]   2> 12329 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10009 (1429684062969135104)]} 0 2
[junit4:junit4]   2> 12333 T177 C9 P42845 /update {version=2&wt=javabin} {add=[9 (1429684062973329408)]} 0 3
[junit4:junit4]   2> 12435 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10010 (1429684063079235584)]} 0 4
[junit4:junit4]   2> 12439 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10 (1429684063084478464)]} 0 4
[junit4:junit4]   2> 12443 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10010 (1429684063087624192)]} 0 3
[junit4:junit4]   2> 12447 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10 (1429684063091818496)]} 0 4
[junit4:junit4]   2> 12486 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[10007 (-1429684063135858688)]} 0 1
[junit4:junit4]   2> 12489 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[7 (-1429684063139004416)]} 0 0
[junit4:junit4]   2> 12490 T176 C9 P42845 /update {version=2&wt=javabin} {delete=[10007 (-1429684063140052992)]} 0 0
[junit4:junit4]   2> 12494 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[7 (-1429684063144247296)]} 0 1
[junit4:junit4]   2> 12499 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10011 (1429684063145295872)]} 0 5
[junit4:junit4]   2> 12501 T156 C8 P45612 /update {version=2&wt=javabin} {add=[11 (1429684063148441600)]} 0 4
[junit4:junit4]   2> 12506 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10011 (1429684063154733056)]} 0 3
[junit4:junit4]   2> 12508 T176 C9 P42845 /update {version=2&wt=javabin} {add=[11 (1429684063155781632)]} 0 3
[junit4:junit4]   2> 12552 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[10009 (-1429684063205064704)]} 0 1
[junit4:junit4]   2> 12554 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[9 (-1429684063207161856)]} 0 1
[junit4:junit4]   2> 12556 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10009 (-1429684063209259008)]} 0 0
[junit4:junit4]   2> 12558 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[9 (-1429684063211356160)]} 0 0
[junit4:junit4]   2> 12564 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10012 (1429684063214501888)]} 0 4
[junit4:junit4]   2> 12566 T156 C8 P45612 /update {version=2&wt=javabin} {add=[12 (1429684063216599040)]} 0 4
[junit4:junit4]   2> 12571 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10012 (1429684063221841920)]} 0 3
[junit4:junit4]   2> 12574 T178 C9 P42845 /update {version=2&wt=javabin} {add=[12 (1429684063224987648)]} 0 4
[junit4:junit4]   2> 12631 T157 C8 P45612 /update {version=2&wt=javabin} {add=[10013 (1429684063286853632)]} 0 2
[junit4:junit4]   2> 12632 T159 C8 P45612 /update {version=2&wt=javabin} {add=[13 (1429684063287902208)]} 0 1
[junit4:junit4]   2> 12636 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10013 (1429684063291047936)]} 0 2
[junit4:junit4]   2> 12637 T176 C9 P42845 /update {version=2&wt=javabin} {add=[13 (1429684063292096512)]} 0 2
[junit4:junit4]   2> 12665 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12666 T164 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:58139",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:58139_",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core":"collection1"}
[junit4:junit4]   2> 12671 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12671 T199 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12671 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12671 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12735 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10014 (1429684063393808384)]} 0 4
[junit4:junit4]   2> 12737 T156 C8 P45612 /update {version=2&wt=javabin} {add=[14 (1429684063393808385)]} 0 5
[junit4:junit4]   2>  C7_STATE=coll:collection1 core:collection1 props:{state=recovering, collection=collection1, base_url=http://127.0.0.1:58139, shard=shard1, node_name=127.0.0.1:58139_, core=collection1}
[junit4:junit4]   2> 12765 T193 C7 P58139 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@27c5c202 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c40d827),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 12766 T193 C7 P58139 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 12770 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10014 (1429684063407439873)]} 0 7
[junit4:junit4]   2> 12771 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[14 (1429684063407439872)]} 0 8
[junit4:junit4]   2> 12771 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10014 (1429684063407439873)]} 0 32
[junit4:junit4]   2> 12772 T177 C9 P42845 /update {version=2&wt=javabin} {add=[14 (1429684063407439872)]} 0 29
[junit4:junit4]   2> 12803 T157 C8 P45612 /update {version=2&wt=javabin} {add=[10015 (1429684063465111552)]} 0 3
[junit4:junit4]   2> 12805 T159 C8 P45612 /update {version=2&wt=javabin} {add=[15 (1429684063465111553)]} 0 5
[junit4:junit4]   2> 12816 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10015 (1429684063472451584)]} 0 3
[junit4:junit4]   2> 12817 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10015 (1429684063472451584)]} 0 11
[junit4:junit4]   2> 12818 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[15 (1429684063474548736)]} 0 3
[junit4:junit4]   2> 12819 T178 C9 P42845 /update {version=2&wt=javabin} {add=[15 (1429684063474548736)]} 0 11
[junit4:junit4]   2> 12850 T179 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {onlyIfLeader=true&checkLive=true&state=recovering&nodeName=127.0.0.1:58139_&version=2&wt=javabin&action=PREPRECOVERY&coreNodeName=3&core=collection1} status=0 QTime=1003 
[junit4:junit4]   2> 12895 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[10010 (-1429684063564726272)]} 0 1
[junit4:junit4]   2> 12896 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[10 (-1429684063565774848)]} 0 0
[junit4:junit4]   2> 12903 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10010 (-1429684063567872000)]} 0 1
[junit4:junit4]   2> 12903 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10 (-1429684063568920576)]} 0 1
[junit4:junit4]   2> 12904 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10010 (-1429684063567872000)]} 0 6
[junit4:junit4]   2> 12904 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[10 (-1429684063568920576)]} 0 5
[junit4:junit4]   2> 12908 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10016 (1429684063577309184)]} 0 2
[junit4:junit4]   2> 12909 T157 C8 P45612 /update {version=2&wt=javabin} {add=[16 (1429684063577309185)]} 0 2
[junit4:junit4]   2> 12919 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10016 (1429684063582552064)]} 0 2
[junit4:junit4]   2> 12920 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10016 (1429684063582552064)]} 0 9
[junit4:junit4]   2> 12921 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[16 (1429684063583600640)]} 0 3
[junit4:junit4]   2> 12922 T176 C9 P42845 /update {version=2&wt=javabin} {add=[16 (1429684063583600640)]} 0 9
[junit4:junit4]   2> 13007 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10017 (1429684063677972480)]} 0 4
[junit4:junit4]   2> 13009 T158 C8 P45612 /update {version=2&wt=javabin} {add=[17 (1429684063680069632)]} 0 5
[junit4:junit4]   2> 13023 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10017 (1429684063687409664)]} 0 5
[junit4:junit4]   2> 13025 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10017 (1429684063687409664)]} 0 14
[junit4:junit4]   2> 13030 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[17 (1429684063689506816)]} 0 5
[junit4:junit4]   2> 13031 T179 C9 P42845 /update {version=2&wt=javabin} {add=[17 (1429684063689506816)]} 0 18
[junit4:junit4]   2> 13069 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10018 (1429684063744032768)]} 0 4
[junit4:junit4]   2> 13073 T157 C8 P45612 /update {version=2&wt=javabin} {add=[18 (1429684063748227072)]} 0 3
[junit4:junit4]   2> 13083 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10018 (1429684063751372800)]} 0 4
[junit4:junit4]   2> 13084 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10018 (1429684063751372800)]} 0 11
[junit4:junit4]   2> 13088 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[18 (1429684063755567104)]} 0 5
[junit4:junit4]   2> 13090 T176 C9 P42845 /update {version=2&wt=javabin} {add=[18 (1429684063755567104)]} 0 13
[junit4:junit4]   2> 13102 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[10011 (-1429684063781781504)]} 0 1
[junit4:junit4]   2> 13107 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[11 (-1429684063787024384)]} 0 0
[junit4:junit4]   2> 13110 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10011 (-1429684063787024384)]} 0 0
[junit4:junit4]   2> 13111 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10011 (-1429684063787024384)]} 0 5
[junit4:junit4]   2> 13114 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[11 (-1429684063792267264)]} 0 0
[junit4:junit4]   2> 13115 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[11 (-1429684063792267264)]} 0 4
[junit4:junit4]   2> 13115 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10019 (1429684063794364416)]} 0 1
[junit4:junit4]   2> 13120 T157 C8 P45612 /update {version=2&wt=javabin} {add=[19 (1429684063798558720)]} 0 2
[junit4:junit4]   2> 13124 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10019 (1429684063798558720)]} 0 2
[junit4:junit4]   2> 13125 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10019 (1429684063798558720)]} 0 7
[junit4:junit4]   2> 13129 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[19 (1429684063803801600)]} 0 2
[junit4:junit4]   2> 13131 T176 C9 P42845 /update {version=2&wt=javabin} {add=[19 (1429684063803801600)]} 0 7
[junit4:junit4]   2> 13221 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[10015 (-1429684063906562048)]} 0 0
[junit4:junit4]   2> 13227 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[15 (-1429684063912853504)]} 0 1
[junit4:junit4]   2> 13227 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10015 (-1429684063909707776)]} 0 0
[junit4:junit4]   2> 13228 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10015 (-1429684063909707776)]} 0 4
[junit4:junit4]   2> 13232 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[15 (-1429684063915999232)]} 0 0
[junit4:junit4]   2> 13233 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[15 (-1429684063915999232)]} 0 3
[junit4:junit4]   2> 13233 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10020 (1429684063917047808)]} 0 2
[junit4:junit4]   2> 13238 T157 C8 P45612 /update {version=2&wt=javabin} {add=[20 (1429684063922290688)]} 0 2
[junit4:junit4]   2> 13241 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10020 (1429684063922290688)]} 0 2
[junit4:junit4]   2> 13242 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10020 (1429684063922290688)]} 0 6
[junit4:junit4]   2> 13247 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[20 (1429684063927533568)]} 0 1
[junit4:junit4]   2> 13248 T178 C9 P42845 /update {version=2&wt=javabin} {add=[20 (1429684063927533568)]} 0 7
[junit4:junit4]   2> 13346 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10021 (1429684064035536896)]} 0 2
[junit4:junit4]   2> 13354 T156 C8 P45612 /update {version=2&wt=javabin} {add=[21 (1429684064043925504)]} 0 2
[junit4:junit4]   2> 13357 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10021 (1429684064040779776)]} 0 2
[junit4:junit4]   2> 13358 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10021 (1429684064040779776)]} 0 9
[junit4:junit4]   2> 13364 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[21 (1429684064050216960)]} 0 2
[junit4:junit4]   2> 13365 T176 C9 P42845 /update {version=2&wt=javabin} {add=[21 (1429684064050216960)]} 0 8
[junit4:junit4]   2> 13414 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[10016 (-1429684064108937216)]} 0 1
[junit4:junit4]   2> 13420 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10016 (-1429684064112082944)]} 0 0
[junit4:junit4]   2> 13420 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[16 (-1429684064115228672)]} 0 0
[junit4:junit4]   2> 13421 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10016 (-1429684064112082944)]} 0 4
[junit4:junit4]   2> 13427 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10022 (1429684064120471552)]} 0 3
[junit4:junit4]   2> 13427 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[16 (-1429684064119422976)]} 0 1
[junit4:junit4]   2> 13428 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[16 (-1429684064119422976)]} 0 5
[junit4:junit4]   2> 13434 T156 C8 P45612 /update {version=2&wt=javabin} {add=[22 (1429684064127811584)]} 0 2
[junit4:junit4]   2> 13439 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10022 (1429684064126763008)]} 0 3
[junit4:junit4]   2> 13441 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10022 (1429684064126763008)]} 0 10
[junit4:junit4]   2> 13447 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[22 (1429684064134103040)]} 0 3
[junit4:junit4]   2> 13448 T176 C9 P42845 /update {version=2&wt=javabin} {add=[22 (1429684064134103040)]} 0 10
[junit4:junit4]   2> 13497 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[10017 (-1429684064195969024)]} 0 0
[junit4:junit4]   2> 13504 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[17 (-1429684064203309056)]} 0 0
[junit4:junit4]   2> 13504 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10017 (-1429684064201211904)]} 0 0
[junit4:junit4]   2> 13513 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10017 (-1429684064201211904)]} 0 12
[junit4:junit4]   2> 13519 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[17 (-1429684064215891968)]} 0 1
[junit4:junit4]   2> 13519 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10023 (1429684064216940544)]} 0 2
[junit4:junit4]   2> 13520 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[17 (-1429684064215891968)]} 0 4
[junit4:junit4]   2> 13526 T157 C8 P45612 /update {version=2&wt=javabin} {add=[23 (1429684064224280576)]} 0 2
[junit4:junit4]   2> 13529 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10023 (1429684064223232000)]} 0 2
[junit4:junit4]   2> 13530 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10023 (1429684064223232000)]} 0 8
[junit4:junit4]   2> 13536 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[23 (1429684064230572032)]} 0 2
[junit4:junit4]   2> 13537 T176 C9 P42845 /update {version=2&wt=javabin} {add=[23 (1429684064230572032)]} 0 7
[junit4:junit4]   2> 13558 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10024 (1429684064258883584)]} 0 2
[junit4:junit4]   2> 13566 T159 C8 P45612 /update {version=2&wt=javabin} {add=[24 (1429684064266223616)]} 0 2
[junit4:junit4]   2> 13568 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10024 (1429684064264126464)]} 0 2
[junit4:junit4]   2> 13570 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10024 (1429684064264126464)]} 0 8
[junit4:junit4]   2> 13576 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[24 (1429684064272515072)]} 0 2
[junit4:junit4]   2> 13577 T179 C9 P42845 /update {version=2&wt=javabin} {add=[24 (1429684064272515072)]} 0 8
[junit4:junit4]   2> 13657 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10025 (1429684064362692608)]} 0 2
[junit4:junit4]   2> 13664 T157 C8 P45612 /update {version=2&wt=javabin} {add=[25 (1429684064368984064)]} 0 2
[junit4:junit4]   2> 13667 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10025 (1429684064367935488)]} 0 2
[junit4:junit4]   2> 13668 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10025 (1429684064367935488)]} 0 8
[junit4:junit4]   2> 13673 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[25 (1429684064374226944)]} 0 3
[junit4:junit4]   2> 13674 T176 C9 P42845 /update {version=2&wt=javabin} {add=[25 (1429684064374226944)]} 0 8
[junit4:junit4]   2> 13677 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[10021 (-1429684064384712704)]} 0 0
[junit4:junit4]   2> 13684 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10021 (-1429684064388907008)]} 0 0
[junit4:junit4]   2> 13684 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[21 (-1429684064392052736)]} 0 0
[junit4:junit4]   2> 13685 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[10021 (-1429684064388907008)]} 0 4
[junit4:junit4]   2> 13691 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[21 (-1429684064396247040)]} 0 1
[junit4:junit4]   2> 13691 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10026 (1429684064398344192)]} 0 2
[junit4:junit4]   2> 13692 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[21 (-1429684064396247040)]} 0 4
[junit4:junit4]   2> 13699 T157 C8 P45612 /update {version=2&wt=javabin} {add=[26 (1429684064406732800)]} 0 2
[junit4:junit4]   2> 13702 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10026 (1429684064403587072)]} 0 2
[junit4:junit4]   2> 13703 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10026 (1429684064403587072)]} 0 8
[junit4:junit4]   2> 13709 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[26 (1429684064411975680)]} 0 2
[junit4:junit4]   2> 13711 T176 C9 P42845 /update {version=2&wt=javabin} {add=[26 (1429684064411975680)]} 0 9
[junit4:junit4]   2> 13721 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[10026 (-1429684064429801472)]} 0 1
[junit4:junit4]   2> 13727 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10026 (-1429684064433995776)]} 0 0
[junit4:junit4]   2> 13727 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[26 (-1429684064437141504)]} 0 0
[junit4:junit4]   2> 13728 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[10026 (-1429684064433995776)]} 0 4
[junit4:junit4]   2> 13732 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10027 (1429684064441335808)]} 0 1
[junit4:junit4]   2> 13737 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[26 (-1429684064440287232)]} 0 0
[junit4:junit4]   2> 13738 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[26 (-1429684064440287232)]} 0 8
[junit4:junit4]   2> 13743 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10027 (1429684064447627264)]} 0 2
[junit4:junit4]   2> 13743 T157 C8 P45612 /update {version=2&wt=javabin} {add=[27 (1429684064452870144)]} 0 2
[junit4:junit4]   2> 13745 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10027 (1429684064447627264)]} 0 8
[junit4:junit4]   2> 13755 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[27 (1429684064459161600)]} 0 3
[junit4:junit4]   2> 13756 T176 C9 P42845 /update {version=2&wt=javabin} {add=[27 (1429684064459161600)]} 0 9
[junit4:junit4]   2> 13801 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10028 (1429684064512638976)]} 0 2
[junit4:junit4]   2> 13810 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10028 (1429684064517881856)]} 0 2
[junit4:junit4]   2> 13811 T179 C9 P42845 /update {version=2&wt=javabin} {add=[10028 (1429684064517881856)]} 0 7
[junit4:junit4]   2> 13811 T156 C8 P45612 /update {version=2&wt=javabin} {add=[28 (1429684064524173312)]} 0 1
[junit4:junit4]   2> 13821 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[28 (1429684064529416192)]} 0 2
[junit4:junit4]   2> 13822 T178 C9 P42845 /update {version=2&wt=javabin} {add=[28 (1429684064529416192)]} 0 8
[junit4:junit4]   2> 13825 T157 C8 P45612 /update {version=2&wt=javabin} {add=[10029 (1429684064537804800)]} 0 2
[junit4:junit4]   2> 13836 T158 C8 P45612 /update {version=2&wt=javabin} {add=[29 (1429684064549339136)]} 0 2
[junit4:junit4]   2> 13838 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10029 (1429684064543047680)]} 0 6
[junit4:junit4]   2> 13839 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10029 (1429684064543047680)]} 0 11
[junit4:junit4]   2> 13843 T141 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 58139
[junit4:junit4]   2> 13844 T141 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1463596936
[junit4:junit4]   2> 13847 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[29 (1429684064555630592)]} 0 3
[junit4:junit4]   2> 13848 T176 C9 P42845 /update {version=2&wt=javabin} {add=[29 (1429684064555630592)]} 0 9
[junit4:junit4]   2> 13942 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10030 (1429684064661536768)]} 0 2
[junit4:junit4]   2> 13950 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10030 (1429684064666779648)]} 0 1
[junit4:junit4]   2> 13951 T157 C8 P45612 /update {version=2&wt=javabin} {add=[30 (1429684064669925376)]} 0 2
[junit4:junit4]   2> 13952 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10030 (1429684064666779648)]} 0 7
[junit4:junit4]   2> 13957 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10031 (1429684064676216832)]} 0 2
[junit4:junit4]   2> 13960 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[30 (1429684064675168256)]} 0 1
[junit4:junit4]   2> 13960 T179 C9 P42845 /update {version=2&wt=javabin} {add=[30 (1429684064675168256)]} 0 6
[junit4:junit4]   2> 13965 T159 C8 P45612 /update {version=2&wt=javabin} {add=[31 (1429684064685654016)]} 0 2
[junit4:junit4]   2> 13965 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10031 (1429684064681459712)]} 0 1
[junit4:junit4]   2> 13966 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10031 (1429684064681459712)]} 0 6
[junit4:junit4]   2> 13979 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[31 (1429684064689848320)]} 0 2
[junit4:junit4]   2> 13980 T176 C9 P42845 /update {version=2&wt=javabin} {add=[31 (1429684064689848320)]} 0 12
[junit4:junit4]   2> 14009 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10032 (1429684064731791360)]} 0 2
[junit4:junit4]   2> 14017 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10032 (1429684064735985664)]} 0 2
[junit4:junit4]   2> 14018 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10032 (1429684064735985664)]} 0 5
[junit4:junit4]   2> 14023 T157 C8 P45612 /update {version=2&wt=javabin} {add=[32 (1429684064746471424)]} 0 2
[junit4:junit4]   2> 14032 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[32 (1429684064750665728)]} 0 2
[junit4:junit4]   2> 14033 T179 C9 P42845 /update {version=2&wt=javabin} {add=[32 (1429684064750665728)]} 0 7
[junit4:junit4]   2> 14047 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[10027 (-1429684064772685824)]} 0 1
[junit4:junit4]   2> 14053 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10027 (-1429684064775831552)]} 0 1
[junit4:junit4]   2> 14054 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[10027 (-1429684064775831552)]} 0 4
[junit4:junit4]   2> 14059 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10033 (1429684064783171584)]} 0 2
[junit4:junit4]   2> 14062 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[27 (-1429684064788414464)]} 0 0
[junit4:junit4]   2> 14067 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10033 (1429684064788414464)]} 0 1
[junit4:junit4]   2> 14068 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10033 (1429684064788414464)]} 0 7
[junit4:junit4]   2> 14069 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[27 (-1429684064792608768)]} 0 1
[junit4:junit4]   2> 14070 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[27 (-1429684064792608768)]} 0 5
[junit4:junit4]   2> 14075 T157 C8 P45612 /update {version=2&wt=javabin} {add=[33 (1429684064799948800)]} 0 2
[junit4:junit4]   2> 14084 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[33 (1429684064805191680)]} 0 2
[junit4:junit4]   2> 14085 T177 C9 P42845 /update {version=2&wt=javabin} {add=[33 (1429684064805191680)]} 0 7
[junit4:junit4]   2> 14120 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10034 (1429684064848183296)]} 0 1
[junit4:junit4]   2> 14128 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10034 (1429684064853426176)]} 0 1
[junit4:junit4]   2> 14129 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10034 (1429684064853426176)]} 0 6
[junit4:junit4]   2> 14138 T159 C8 P45612 /update {version=2&wt=javabin} {add=[34 (1429684064866009088)]} 0 2
[junit4:junit4]   2> 14147 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[34 (1429684064871251968)]} 0 2
[junit4:junit4]   2> 14148 T179 C9 P42845 /update {version=2&wt=javabin} {add=[34 (1429684064871251968)]} 0 7
[junit4:junit4]   2> 14176 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14177 T164 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:58139",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:58139_",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core":"collection1"}
[junit4:junit4]   2> 14191 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14192 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14192 T199 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14191 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14230 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[10028 (-1429684064964575232)]} 0 0
[junit4:junit4]   2> 14233 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10028 (-1429684064967720960)]} 0 0
[junit4:junit4]   2> 14237 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10035 (1429684064970866688)]} 0 1
[junit4:junit4]   2> 14241 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10035 (1429684064975060992)]} 0 1
[junit4:junit4]   2> 14249 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[28 (-1429684064984498176)]} 0 0
[junit4:junit4]   2> 14253 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[28 (-1429684064987643904)]} 0 1
[junit4:junit4]   2> 14257 T157 C8 P45612 /update {version=2&wt=javabin} {add=[35 (1429684064991838208)]} 0 2
[junit4:junit4]   2> 14261 T178 C9 P42845 /update {version=2&wt=javabin} {add=[35 (1429684064996032512)]} 0 1
[junit4:junit4]   2> 14311 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[10030 (-1429684065049509888)]} 0 0
[junit4:junit4]   2> 14316 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[10030 (-1429684065053704192)]} 0 1
[junit4:junit4]   2> 14320 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10036 (1429684065057898496)]} 0 1
[junit4:junit4]   2> 14324 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10036 (1429684065062092800)]} 0 2
[junit4:junit4]   2> 14331 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[30 (-1429684065070481408)]} 0 0
[junit4:junit4]   2> 14334 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[30 (-1429684065073627136)]} 0 0
[junit4:junit4]   2> 14338 T157 C8 P45612 /update {version=2&wt=javabin} {add=[36 (1429684065076772864)]} 0 1
[junit4:junit4]   2> 14342 T178 C9 P42845 /update {version=2&wt=javabin} {add=[36 (1429684065080967168)]} 0 1
[junit4:junit4]   2> 14345 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10037 (1429684065084112896)]} 0 1
[junit4:junit4]   2> 14350 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10037 (1429684065089355776)]} 0 2
[junit4:junit4]   2> 14365 T158 C8 P45612 /update {version=2&wt=javabin} {add=[37 (1429684065105084416)]} 0 2
[junit4:junit4]   2> 14369 T176 C9 P42845 /update {version=2&wt=javabin} {add=[37 (1429684065109278720)]} 0 2
[junit4:junit4]   2> 14382 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[10035 (-1429684065123958784)]} 0 1
[junit4:junit4]   2> 14384 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[10035 (-1429684065126055936)]} 0 0
[junit4:junit4]   2> 14388 T157 C8 P45612 /update {version=2&wt=javabin} {add=[10038 (1429684065129201664)]} 0 1
[junit4:junit4]   2> 14393 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10038 (1429684065133395968)]} 0 2
[junit4:junit4]   2> 14401 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[35 (-1429684065142833152)]} 0 1
[junit4:junit4]   2> 14405 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[35 (-1429684065148076032)]} 0 1
[junit4:junit4]   2> 14411 T158 C8 P45612 /update {version=2&wt=javabin} {add=[38 (1429684065152270336)]} 0 2
[junit4:junit4]   2> 14416 T176 C9 P42845 /update {version=2&wt=javabin} {add=[38 (1429684065157513216)]} 0 2
[junit4:junit4]   2> 14492 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[10036 (-1429684065239302144)]} 0 0
[junit4:junit4]   2> 14495 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[10036 (-1429684065242447872)]} 0 0
[junit4:junit4]   2> 14499 T157 C8 P45612 /update {version=2&wt=javabin} {add=[10039 (1429684065244545024)]} 0 2
[junit4:junit4]   2> 14502 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10039 (1429684065248739328)]} 0 1
[junit4:junit4]   2> 14515 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[36 (-1429684065263419392)]} 0 0
[junit4:junit4]   2> 14518 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[36 (-1429684065266565120)]} 0 0
[junit4:junit4]   2> 14522 T158 C8 P45612 /update {version=2&wt=javabin} {add=[39 (1429684065269710848)]} 0 2
[junit4:junit4]   2> 14526 T176 C9 P42845 /update {version=2&wt=javabin} {add=[39 (1429684065273905152)]} 0 1
[junit4:junit4]   2> 14565 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10040 (1429684065314799616)]} 0 2
[junit4:junit4]   2> 14569 T179 C9 P42845 /update {version=2&wt=javabin} {add=[10040 (1429684065318993920)]} 0 1
[junit4:junit4]   2> 14584 T157 C8 P45612 /update {version=2&wt=javabin} {add=[10041 (1429684065334722560)]} 0 2
[junit4:junit4]   2> 14588 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10041 (1429684065338916864)]} 0 1
[junit4:junit4]   2> 14588 T156 C8 P45612 /update {version=2&wt=javabin} {add=[40 (1429684065338916864)]} 0 1
[junit4:junit4]   2> 14592 T177 C9 P42845 /update {version=2&wt=javabin} {add=[40 (1429684065343111168)]} 0 1
[junit4:junit4]   2> 14608 T158 C8 P45612 /update {version=2&wt=javabin} {add=[41 (1429684065358839808)]} 0 2
[junit4:junit4]   2> 14612 T176 C9 P42845 /update {version=2&wt=javabin} {add=[41 (1429684065364082688)]} 0 2
[junit4:junit4]   2> 14651 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10042 (1429684065404977152)]} 0 1
[junit4:junit4]   2> 14655 T179 C9 P42845 /update {version=2&wt=javabin} {add=[10042 (1429684065409171456)]} 0 2
[junit4:junit4]   2> 14675 T156 C8 P45612 /update {version=2&wt=javabin} {add=[42 (1429684065429094400)]} 0 2
[junit4:junit4]   2> 14679 T177 C9 P42845 /update {version=2&wt=javabin} {add=[42 (1429684065434337280)]} 0 2
[junit4:junit4]   2> 14746 T157 C8 P45612 /update {version=2&wt=javabin} {add=[10043 (1429684065504591872)]} 0 1
[junit4:junit4]   2> 14750 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10043 (1429684065508786176)]} 0 1
[junit4:junit4]   2> 14767 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[10043 (-1429684065526611968)]} 0 1
[junit4:junit4]   2> 14769 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10043 (-1429684065529757696)]} 0 0
[junit4:junit4]   2> 14770 T156 C8 P45612 /update {version=2&wt=javabin} {add=[43 (1429684065529757696)]} 0 1
[junit4:junit4]   2> 14773 T179 C9 P42845 /update {version=2&wt=javabin} {add=[43 (1429684065532903424)]} 0 1
[junit4:junit4]   2> 14774 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10044 (1429684065532903424)]} 0 2
[junit4:junit4]   2> 14778 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10044 (1429684065538146304)]} 0 1
[junit4:junit4]   2> 14790 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[43 (-1429684065551777792)]} 0 0
[junit4:junit4]   2> 14793 T176 C9 P42845 /update {version=2&wt=javabin} {delete=[43 (-1429684065554923520)]} 0 0
[junit4:junit4]   2> 14797 T159 C8 P45612 /update {version=2&wt=javabin} {add=[44 (1429684065558069248)]} 0 1
[junit4:junit4]   2> 14801 T178 C9 P42845 /update {version=2&wt=javabin} {add=[44 (1429684065562263552)]} 0 1
[junit4:junit4]   2> 14843 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10045 (1429684065606303744)]} 0 1
[junit4:junit4]   2> 14847 T179 C9 P42845 /update {version=2&wt=javabin} {add=[10045 (1429684065610498048)]} 0 1
[junit4:junit4]   2>  C7_STATE=coll:collection1 core:collection1 props:{state=down, collection=collection1, base_url=http://127.0.0.1:58139, shard=shard1, node_name=127.0.0.1:58139_, core=collection1}
[junit4:junit4]   2> 14851 T202 C7 P58139 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:42845/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 14851 T202 C7 P58139 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58139 START replicas=[http://127.0.0.1:42845/collection1/] nUpdates=100
[junit4:junit4]   2> 14852 T202 C7 P58139 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 14853 T202 C7 P58139 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 14853 T202 C7 P58139 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 14853 T202 C7 P58139 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 14854 T202 C7 P58139 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4]   2> 14854 T202 C7 P58139 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:42845/collection1/. core=collection1
[junit4:junit4]   2> 14854 T202 C7 P58139 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 14854 T176 C9 P42845 REQ /get {getVersions=100&version=2&wt=javabin&qt=/get&distrib=false} status=0 QTime=0 
[junit4:junit4]   2> 14859 T178 C9 P42845 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 14866 T158 C8 P45612 /update {version=2&wt=javabin} {add=[45 (1429684065629372416)]} 0 2
[junit4:junit4]   2> 14871 T179 C9 P42845 /update {version=2&wt=javabin} {add=[45 (1429684065634615296)]} 0 2
[junit4:junit4]   2> 14915 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[10044 (-1429684065682849792)]} 0 0
[junit4:junit4]   2> 14919 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[10044 (-1429684065687044096)]} 0 1
[junit4:junit4]   2> 14919 T178 C9 P42845 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5640b971 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3e5045db),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5640b971 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3e5045db),segFN=segments_2,generation=2,filenames=[_0_Direct_0.doc, _0_Lucene41_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.pos, segments_2, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _0_Asserting_0.tip, _0_1.del, _0_Direct_0.tim, _0_Direct_0.pos, _0_MockFixedIntBlock_0.doc, _0_MockFixedIntBlock_0.frq, _0.si, _0_Lucene41_0.tim, _0_MockFixedIntBlock_0.pyl, _0_Direct_0.tip, _0.fnm, _0_Lucene41_0.doc, _0.fdt, _0.nvd, _0_Lucene41_0.tip, _0_Asserting_0.pos, _0.fdx, _0.nvm, _0_Asserting_0.doc]
[junit4:junit4]   2> 14920 T178 C9 P42845 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_Direct_0.doc, _0_Lucene41_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.pos, segments_2, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _0_Asserting_0.tip, _0_1.del, _0_Direct_0.tim, _0_Direct_0.pos, _0_MockFixedIntBlock_0.doc, _0_MockFixedIntBlock_0.frq, _0.si, _0_Lucene41_0.tim, _0_MockFixedIntBlock_0.pyl, _0_Direct_0.tip, _0.fnm, _0_Lucene41_0.doc, _0.fdt, _0.nvd, _0_Lucene41_0.tip, _0_Asserting_0.pos, _0.fdx, _0.nvm, _0_Asserting_0.doc]
[junit4:junit4]   2> 14924 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10046 (1429684065690189824)]} 0 2
[junit4:junit4]   2> 14928 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10046 (1429684065695432704)]} 0 2
[junit4:junit4]   2> 14939 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[44 (-1429684065708015616)]} 0 0
[junit4:junit4]   2> 14942 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[44 (-1429684065711161344)]} 0 0
[junit4:junit4]   2> 14947 T158 C8 P45612 /update {version=2&wt=javabin} {add=[46 (1429684065714307072)]} 0 2
[junit4:junit4]   2> 14951 T177 C9 P42845 /update {version=2&wt=javabin} {add=[46 (1429684065719549952)]} 0 2
[junit4:junit4]   2> 14955 T178 C9 P42845 oass.SolrIndexSearcher.<init> Opening Searcher@103021f5 realtime
[junit4:junit4]   2> 14956 T178 C9 P42845 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 14956 T178 C9 P42845 /update {waitSearcher=true&commit_end_point=true&version=2&wt=javabin&softCommit=false&openSearcher=false&commit=true} {commit=} 0 97
[junit4:junit4]   2> 14974 T202 C7 P58139 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 14974 T202 C7 P58139 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4]   2> 14979 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[10045 (-1429684065748910080)]} 0 1
[junit4:junit4]   2> 14979 T202 C7 P58139 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 14982 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[10045 (-1429684065753104384)]} 0 1
[junit4:junit4]   2> 14984 T179 C9 P42845 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 14985 T179 C9 P42845 REQ /replication {command=indexversion&version=2&qt=/replication&wt=javabin} status=0 QTime=4 
[junit4:junit4]   2> 14986 T202 C7 P58139 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 14986 T202 C7 P58139 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 14987 T202 C7 P58139 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 14986 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10047 (1429684065756250112)]} 0 1
[junit4:junit4]   2> 14989 T178 C9 P42845 REQ /replication {command=filelist&generation=2&version=2&qt=/replication&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 14991 T202 C7 P58139 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 27
[junit4:junit4]   2> 14991 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10047 (1429684065760444416)]} 0 2
[junit4:junit4]   2> 14991 T202 C7 P58139 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/index.20130316235614093 forceNew: false
[junit4:junit4]   2> 14992 T202 C7 P58139 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@34a32a11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19085b9d) fullCopy=true
[junit4:junit4]   2> 14998 T177 C9 P42845 REQ /replication {command=filecontent&file=_0_Direct_0.doc&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=2 
[junit4:junit4]   2> 15001 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[45 (-1429684065773027328)]} 0 0
[junit4:junit4]   2> 15004 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[45 (-1429684065776173056)]} 0 0
[junit4:junit4]   2> 15005 T178 C9 P42845 REQ /replication {command=filecontent&file=_0_Lucene41_0.pos&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15008 T176 C9 P42845 REQ /replication {command=filecontent&file=_0_MockFixedIntBlock_0.pos&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15009 T158 C8 P45612 /update {version=2&wt=javabin} {add=[47 (1429684065779318784)]} 0 2
[junit4:junit4]   2> 15011 T177 C9 P42845 REQ /replication {command=filecontent&file=_0_Asserting_0.tim&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15013 T179 C9 P42845 /update {version=2&wt=javabin} {add=[47 (1429684065784561664)]} 0 2
[junit4:junit4]   2> 15014 T178 C9 P42845 REQ /replication {command=filecontent&file=segments_2&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15018 T176 C9 P42845 REQ /replication {command=filecontent&file=_0_MockFixedIntBlock_0.tii&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15022 T177 C9 P42845 REQ /replication {command=filecontent&file=_0_MockFixedIntBlock_0.tib&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=1 
[junit4:junit4]   2> 15025 T179 C9 P42845 REQ /replication {command=filecontent&file=_0_MockFixedIntBlock_0.skp&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15029 T178 C9 P42845 REQ /replication {command=filecontent&file=_0_Asserting_0.tip&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15032 T176 C9 P42845 REQ /replication {command=filecontent&file=_0_1.del&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15035 T177 C9 P42845 REQ /replication {command=filecontent&file=_0_Direct_0.tim&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15038 T179 C9 P42845 REQ /replication {command=filecontent&file=_0_MockFixedIntBlock_0.doc&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15042 T178 C9 P42845 REQ /replication {command=filecontent&file=_0_Direct_0.pos&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=1 
[junit4:junit4]   2> 15045 T176 C9 P42845 REQ /replication {command=filecontent&file=_0.si&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15048 T177 C9 P42845 REQ /replication {command=filecontent&file=_0_MockFixedIntBlock_0.frq&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15052 T179 C9 P42845 REQ /replication {command=filecontent&file=_0_Lucene41_0.tim&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15055 T178 C9 P42845 REQ /replication {command=filecontent&file=_0_MockFixedIntBlock_0.pyl&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=1 
[junit4:junit4]   2> 15056 T202 C7 P58139 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {size=0, name=_0_MockFixedIntBlock_0.pyl}
[junit4:junit4]   2> 15059 T176 C9 P42845 REQ /replication {command=filecontent&file=_0_Direct_0.tip&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=1 
[junit4:junit4]   2> 15063 T177 C9 P42845 REQ /replication {command=filecontent&file=_0_Lucene41_0.doc&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=1 
[junit4:junit4]   2> 15067 T179 C9 P42845 REQ /replication {command=filecontent&file=_0.fnm&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15071 T178 C9 P42845 REQ /replication {command=filecontent&file=_0.fdt&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=1 
[junit4:junit4]   2> 15074 T176 C9 P42845 REQ /replication {command=filecontent&file=_0.nvd&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15075 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[10046 (-1429684065850621952)]} 0 0
[junit4:junit4]   2> 15078 T177 C9 P42845 REQ /replication {command=filecontent&file=_0_Lucene41_0.tip&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15078 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[10046 (-1429684065853767680)]} 0 0
[junit4:junit4]   2> 15081 T176 C9 P42845 REQ /replication {command=filecontent&file=_0_Asserting_0.pos&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15082 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10048 (1429684065856913408)]} 0 1
[junit4:junit4]   2> 15086 T178 C9 P42845 REQ /replication {command=filecontent&file=_0.fdx&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=1 
[junit4:junit4]   2> 15087 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10048 (1429684065862156288)]} 0 2
[junit4:junit4]   2> 15090 T179 C9 P42845 REQ /replication {command=filecontent&file=_0.nvm&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=1 
[junit4:junit4]   2> 15093 T176 C9 P42845 REQ /replication {command=filecontent&file=_0_Asserting_0.doc&generation=2&wt=filestream&qt=/replication&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 15094 T202 C7 P58139 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 15094 T202 C7 P58139 oash.SnapPuller.modifyIndexProps New index installed. Updating index properties... index=index.20130316235614093
[junit4:junit4]   2> 15097 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[46 (-1429684065873690624)]} 0 0
[junit4:junit4]   2> 15100 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[46 (-1429684065876836352)]} 0 0
[junit4:junit4]   2> 15104 T202 C7 P58139 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 15105 T159 C8 P45612 /update {version=2&wt=javabin} {add=[48 (1429684065879982080)]} 0 2
[junit4:junit4]   2> 15105 T202 C7 P58139 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 15105 T202 C7 P58139 oasu.DefaultSolrCoreState.newIndexWriter Rollback old IndexWriter... core=collection1
[junit4:junit4]   2> 15107 T202 C7 P58139 oasc.SolrCore.getNewIndexDir New index directory detected: old=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/index/ new=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/index.20130316235614093
[junit4:junit4]   2> 15109 T177 C9 P42845 /update {version=2&wt=javabin} {add=[48 (1429684065885224960)]} 0 1
[junit4:junit4]   2> 15110 T202 C7 P58139 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34a32a11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19085b9d),segFN=segments_2,generation=2,filenames=[_0_Direct_0.doc, _0_Lucene41_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.pos, segments_2, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _0_Asserting_0.tip, _0_1.del, _0_Direct_0.tim, _0_Direct_0.pos, _0_MockFixedIntBlock_0.doc, _0.si, _0_MockFixedIntBlock_0.frq, _0_Lucene41_0.tim, _0_MockFixedIntBlock_0.pyl, _0_Direct_0.tip, _0_Lucene41_0.doc, _0.fnm, _0.fdt, _0.nvd, _0_Lucene41_0.tip, _0_Asserting_0.pos, _0.fdx, _0.nvm, _0_Asserting_0.doc]
[junit4:junit4]   2> 15110 T202 C7 P58139 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0_Direct_0.doc, _0_Lucene41_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.pos, segments_2, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _0_Asserting_0.tip, _0_1.del, _0_Direct_0.tim, _0_Direct_0.pos, _0_MockFixedIntBlock_0.doc, _0.si, _0_MockFixedIntBlock_0.frq, _0_Lucene41_0.tim, _0_MockFixedIntBlock_0.pyl, _0_Direct_0.tip, _0_Lucene41_0.doc, _0.fnm, _0.fdt, _0.nvd, _0_Lucene41_0.tip, _0_Asserting_0.pos, _0.fdx, _0.nvm, _0_Asserting_0.doc]
[junit4:junit4]   2> 15111 T202 C7 P58139 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 15115 T202 C7 P58139 oass.SolrIndexSearcher.<init> Opening Searcher@3ebc368b main
[junit4:junit4]   2> 15116 T201 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ebc368b main{StandardDirectoryReader(segments_2:4:nrt _0(4.3):C89/36)}
[junit4:junit4]   2> 15116 T202 C7 P58139 oash.SnapPuller$2.preClose removing old index files MockDirWrapper(org.apache.lucene.store.RAMDirectory@27c5c202 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c40d827)
[junit4:junit4]   2> 15117 T202 C7 P58139 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/index
[junit4:junit4]   2> 15117 T202 C7 P58139 oash.SnapPuller$2.postClose removing old index directory MockDirWrapper(org.apache.lucene.store.RAMDirectory@27c5c202 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c40d827)
[junit4:junit4]   2> 15119 T202 C7 P58139 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4]   2> 15119 T210 C7 P58139 oasu.UpdateLog$LogReplayer.doReplay WARNING Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363452959107/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=4696
[junit4:junit4]   2> 15120 T210 C7 P58139 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 15122 T210 C7 P58139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34a32a11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19085b9d),segFN=segments_2,generation=2,filenames=[_0_Direct_0.doc, _0_Lucene41_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.pos, segments_2, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _0_Asserting_0.tip, _0_1.del, _0_Direct_0.tim, _0_Direct_0.pos, _0_MockFixedIntBlock_0.doc, _0.si, _0_MockFixedIntBlock_0.frq, _0_Lucene41_0.tim, _0_MockFixedIntBlock_0.pyl, _0_Direct_0.tip, _0_Lucene41_0.doc, _0.fnm, _0.fdt, _0.nvd, _0_Lucene41_0.tip, _0_Asserting_0.pos, _0.fdx, _0.nvm, _0_Asserting_0.doc]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34a32a11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19085b9d),segFN=segments_3,generation=3,filenames=[_0_Direct_0.doc, _0_Lucene41_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.pos, segments_3, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _0_Asserting_0.tip, _0_1.del, _0_Direct_0.tim, _0_Direct_0.pos, _0_MockFixedIntBlock_0.doc, _0.si, _0_MockFixedIntBlock_0.frq, _0_Lucene41_0.tim, _0_MockFixedIntBlock_0.pyl, _0_Direct_0.tip, _0_Lucene41_0.doc, _0.fnm, _0.fdt, _0.nvd, _0_Lucene41_0.tip, _0_Asserting_0.pos, _0.fdx, _0.nvm, _0_Asserting_0.doc]
[junit4:junit4]   2> 15122 T210 C7 P58139 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0_Direct_0.doc, _0_Lucene41_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.pos, segments_3, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _0_Asserting_0.tip, _0_1.del, _0_Direct_0.tim, _0_Direct_0.pos, _0_MockFixedIntBlock_0.doc, _0.si, _0_MockFixedIntBlock_0.frq, _0_Lucene41_0.tim, _0_MockFixedIntBlock_0.pyl, _0_Direct_0.tip, _0_Lucene41_0.doc, _0.fnm, _0.fdt, _0.nvd, _0_Lucene41_0.tip, _0_Asserting_0.pos, _0.fdx, _0.nvm, _0_Asserting_0.doc]
[junit4:junit4]   2> 15123 T210 C7 P58139 oass.SolrIndexSearcher.<init> Opening Searcher@31b73442 main
[junit4:junit4]   2> 15123 T210 C7 P58139 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 15124 T201 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@31b73442 main{StandardDirectoryReader(segments_2:4:nrt _0(4.3):C89/36)}
[junit4:junit4]   2> 15124 T210 C7 P58139 oasu.UpdateLog$LogReplayer.run WARNING Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=4696}
[junit4:junit4]   2> 15124 T202 C7 P58139 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 15125 T202 C7 P58139 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 15125 T202 C7 P58139 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15129 T202 C7 P58139 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 15192 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10049 (1429684065971208192)]} 0 3
[junit4:junit4]   2> 15198 T179 C9 P42845 /update {version=2&wt=javabin} {add=[10049 (1429684065978548224)]} 0 2
[junit4:junit4]   2> 15212 T158 C8 P45612 /update {version=2&wt=javabin} {add=[49 (1429684065993228288)]} 0 1
[junit4:junit4]   2> 15216 T176 C9 P42845 /update {version=2&wt=javabin} {add=[49 (1429684065997422592)]} 0 1
[junit4:junit4]   2> 15248 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10050 (1429684066030977024)]} 0 1
[junit4:junit4]   2> 15252 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10050 (1429684066035171328)]} 0 1
[junit4:junit4]   2> 15265 T157 C8 P45612 /update {version=2&wt=javabin} {add=[50 (1429684066048802816)]} 0 1
[junit4:junit4]   2> 15270 T177 C9 P42845 /update {version=2&wt=javabin} {add=[50 (1429684066054045696)]} 0 2
[junit4:junit4]   2> 15296 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10051 (1429684066081308672)]} 0 1
[junit4:junit4]   2> 15300 T179 C9 P42845 /update {version=2&wt=javabin} {add=[10051 (1429684066085502976)]} 0 1
[junit4:junit4]   2> 15315 T158 C8 P45612 /update {version=2&wt=javabin} {add=[51 (1429684066100183040)]} 0 2
[junit4:junit4]   2> 15319 T176 C9 P42845 /update {version=2&wt=javabin} {add=[51 (1429684066105425920)]} 0 1
[junit4:junit4]   2> 15392 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10052 (1429684066181971968)]} 0 2
[junit4:junit4]   2> 15396 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10052 (1429684066186166272)]} 0 1
[junit4:junit4]   2> 15410 T157 C8 P45612 /update {version=2&wt=javabin} {add=[52 (1429684066200846336)]} 0 1
[junit4:junit4]   2> 15414 T177 C9 P42845 /update {version=2&wt=javabin} {add=[52 (1429684066205040640)]} 0 1
[junit4:junit4]   2> 15496 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10053 (1429684066291023872)]} 0 2
[junit4:junit4]   2> 15500 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10053 (1429684066295218176)]} 0 1
[junit4:junit4]   2> 15513 T159 C8 P45612 /update {version=2&wt=javabin} {add=[53 (1429684066308849664)]} 0 1
[junit4:junit4]   2> 15517 T177 C9 P42845 /update {version=2&wt=javabin} {add=[53 (1429684066313043968)]} 0 1
[junit4:junit4]   2> 15539 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[10051 (-1429684066337161216)]} 0 0
[junit4:junit4]   2> 15542 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[10051 (-1429684066340306944)]} 0 0
[junit4:junit4]   2> 15546 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10054 (1429684066343452672)]} 0 1
[junit4:junit4]   2> 15549 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10054 (1429684066347646976)]} 0 1
[junit4:junit4]   2> 15556 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[51 (-1429684066354987008)]} 0 0
[junit4:junit4]   2> 15559 T176 C9 P42845 /update {version=2&wt=javabin} {delete=[51 (-1429684066358132736)]} 0 0
[junit4:junit4]   2> 15563 T159 C8 P45612 /update {version=2&wt=javabin} {add=[54 (1429684066361278464)]} 0 1
[junit4:junit4]   2> 15566 T177 C9 P42845 /update {version=2&wt=javabin} {add=[54 (1429684066365472768)]} 0 1
[junit4:junit4]   2> 15596 T157 C8 P45612 /update {version=2&wt=javabin} {add=[10055 (1429684066395881472)]} 0 2
[junit4:junit4]   2> 15600 T179 C9 P42845 /update {version=2&wt=javabin} {add=[10055 (1429684066400075776)]} 0 1
[junit4:junit4]   2> 15613 T158 C8 P45612 /update {version=2&wt=javabin} {add=[55 (1429684066413707264)]} 0 2
[junit4:junit4]   2> 15617 T178 C9 P42845 /update {version=2&wt=javabin} {add=[55 (1429684066417901568)]} 0 1
[junit4:junit4]   2> 15682 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[10054 (-1429684066486059008)]} 0 1
[junit4:junit4]   2> 15686 T176 C9 P42845 /update {version=2&wt=javabin} {delete=[10054 (-1429684066491301888)]} 0 1
[junit4:junit4]   2> 15692 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10056 (1429684066495496192)]} 0 2
[junit4:junit4]   2> 15698 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[54 (-1429684066502836224)]} 0 1
[junit4:junit4]   2> 15698 T164 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 15699 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10056 (1429684066502836224)]} 0 3
[junit4:junit4]   2> 15700 T164 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "core_node_name":"3",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:58139",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:58139_",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core":"collection1"}
[junit4:junit4]   2> 15702 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[54 (-1429684066508079104)]} 0 1
[junit4:junit4]   2> 15708 T158 C8 P45612 /update {version=2&wt=javabin} {add=[56 (1429684066512273408)]} 0 3
[junit4:junit4]   2> 15712 T178 C9 P42845 /update {version=2&wt=javabin} {add=[56 (1429684066517516288)]} 0 1
[junit4:junit4]   2> 15741 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 15741 T199 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 15741 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 15741 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 15763 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[10055 (-1429684066572042240)]} 0 1
[junit4:junit4]   2>  C7_STATE=coll:collection1 core:collection1 props:{state=active, collection=collection1, base_url=http://127.0.0.1:58139, shard=shard1, node_name=127.0.0.1:58139_, core=collection1}
[junit4:junit4]   2> 15769 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10055 (-1429684066575187968)]} 0 1
[junit4:junit4]   2> 15770 T176 C9 P42845 /update {version=2&wt=javabin} {delete=[10055 (-1429684066575187968)]} 0 4
[junit4:junit4]   2> 15774 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10057 (1429684066582528000)]} 0 2
[junit4:junit4]   2> 15775 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[55 (-1429684066583576576)]} 0 0
[junit4:junit4]   2> 15780 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[55 (-1429684066586722304)]} 0 1
[junit4:junit4]   2> 15781 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[55 (-1429684066586722304)]} 0 4
[junit4:junit4]   2> 15782 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10057 (1429684066585673728)]} 0 2
[junit4:junit4]   2> 15783 T177 C9 P42845 /update {version=2&wt=javabin} {add=[10057 (1429684066585673728)]} 0 7
[junit4:junit4]   2> 15787 T156 C8 P45612 /update {version=2&wt=javabin} {add=[57 (1429684066595110912)]} 0 1
[junit4:junit4]   2> 15794 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[57 (1429684066600353792)]} 0 1
[junit4:junit4]   2> 15795 T178 C9 P42845 /update {version=2&wt=javabin} {add=[57 (1429684066600353792)]} 0 6
[junit4:junit4]   2> 15807 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10058 (1429684066617131008)]} 0 1
[junit4:junit4]   2> 15819 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10058 (1429684066623422464)]} 0 2
[junit4:junit4]   2> 15820 T159 C8 P45612 /update {version=2&wt=javabin} {add=[58 (1429684066629713920)]} 0 3
[junit4:junit4]   2> 15821 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10058 (1429684066623422464)]} 0 10
[junit4:junit4]   2> 15831 T194 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[58 (1429684066637053952)]} 0 2
[junit4:junit4]   2> 15831 T179 C9 P42845 /update {version=2&wt=javabin} {add=[58 (1429684066637053952)]} 0 7
[junit4:junit4]   2> 15849 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[10056 (-1429684066662219776)]} 0 0
[junit4:junit4]   2> 15857 T195 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[10056 (-1429684066666414080)]} 0 1
[junit4:junit4]   2> 15858 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[10056 (-1429684066666414080)]} 0 5
[junit4:junit4]   2> 15859 T141 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 15860 T141 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15864 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[56 (-1429684066677948416)]} 0 1
[junit4:junit4]   2> 15866 T158 C8 P45612 /update {version=2&wt=javabin} {add=[10059 (1429684066677948417)]} 0 3
[junit4:junit4]   2> 15866 T141 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3b84b706
[junit4:junit4]   2> 15871 T192 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {delete=[56 (-1429684066683191296)]} 0 0
[junit4:junit4]   2> 15872 T192 C7 P58139 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3b84b706
[junit4:junit4]   2> 15874 T193 C7 P58139 /update {version=2&wt=javabin&distrib.from=http://127.0.0.1:42845/collection1/&update.distrib=FROMLEADER} {add=[10059 (1429684066684239872)]} 0 1
[junit4:junit4]   2> 15874 T193 C7 P58139 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3b84b706
[junit4:junit4]   2> 15875 T192 C7 P58139 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: Failed to unregister info bean: Searcher@31b73442 main
[junit4:junit4]   2> 		at org.apache.solr.core.JmxMonitoredMap.unregister(JmxMonitoredMap.java:178)
[junit4:junit4]   2> 		at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:116)
[junit4:junit4]   2> 		at org.apache.solr.core.SolrCore.close(SolrCore.java:970)
[junit4:junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:372)
[junit4:junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4]   2> 		at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4]   2> 		at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]   2> 		at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 	Caused by: javax.management.InstanceNotFoundException: solr/collection1:id=org.apache.solr.search.SolrIndexSearcher,type=Searcher@31b73442 main
[junit4:junit4]   2> 		at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
[junit4:junit4]   2> 		at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
[junit4:junit4]   2> 		at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
[junit4:junit4]   2> 		at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:536)
[junit4:junit4]   2> 		at org.apache.solr.core.JmxMonitoredMap.unregister(JmxMonitoredMap.java:175)
[junit4:junit4]   2> 		... 29 more
[junit4:junit4]   2> 	
[junit4:junit4]   2> 15875 T141 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=5,adds=5,deletesById=4,deletesByQuery=0,errors=0,cumulative_adds=47,cumulative_deletesById=20,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 15876 T192 C7 P58139 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=5,adds=5,deletesById=4,deletesByQuery=0,errors=0,cumulative_adds=47,cumulative_deletesById=20,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 15876 T141 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 15876 T193 C7 P58139 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: Failed to unregister info bean: core
[junit4:junit4]   2> 		at org.apache.solr.core.JmxMonitoredMap.unregister(JmxMonitoredMap.java:178)
[junit4:junit4]   2> 		at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:116)
[junit4:junit4]   2> 		at org.apache.solr.core.SolrCore.close(SolrCore.java:970)
[junit4:junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:372)
[junit4:junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4]   2> 		at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4]   2> 		at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]   2> 		at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 	Caused by: javax.management.InstanceNotFoundException: solr/collection1:id=collection1,type=core
[junit4:junit4]   2> 		at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
[junit4:junit4]   2> 		at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
[junit4:junit4]   2> 		at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
[junit4:junit4]   2> 		at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:536)
[junit4:junit4]   2> 		at org.apache.solr.core.JmxMonitoredMap.unregister(JmxMonitoredMap.java:175)
[junit4:junit4]   2> 		... 29 more
[junit4:junit4]   2> 	
[junit4:junit4]   2> 15877 T192 C7 P58139 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 15877 T141 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 15877 T193 C7 P58139 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=5,deletesById=4,deletesByQuery=0,errors=0,cumulative_adds=47,cumulative_deletesById=20,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 15877 T141 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 15886 T141 C7 P58139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34a32a11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19085b9d),segFN=segments_3,generation=3,filenames=[_0_Direct_0.doc, _0_Lucene41_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.pos, segments_3, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _0_Asserting_0.tip, _0_1.del, _0_Direct_0.tim, _0_Direct_0.pos, _0_MockFixedIntBlock_0.doc, _0.si, _0_MockFixedIntBlock_0.frq, _0_Lucene41_0.tim, _0_MockFixedIntBlock_0.pyl, _0_Direct_0.tip, _0_Lucene41_0.doc, _0.fnm, _0.fdt, _0.nvd, _0_Lucene41_0.tip, _0_Asserting_0.pos, _0.fdx, _0.nvm, _0_Asserting_0.doc]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@34a32a11 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19085b9d),segFN=segments_4,generation=4,filenames=[_0_Lucene41_0.pos, _2_Direct_0.doc, _2_MockFixedIntBlock_0.tib, _2.fdt, _0_MockFixedIntBlock_0.tib, _2.fnm, _0_1.del, _0_Direct_0.tim, _2.nvm, _2_MockFixedIntBlock_0.frq, _0.si, _0_Lucene41_0.tim, _2_Lucene41_0.tip, _0_Direct_0.tip, _0.fnm, _0_Lucene41_0.doc, _2_Direct_0.tip, _0_Lucene41_0.tip, _2.fdx, _0_Asserting_0.pos, _2_Direct_0.tim, _0.nvm, _2_Direct_0.pos, _0_Direct_0.doc, _2_MockFixedIntBlock_0.pos, _0_MockFixedIntBlock_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _2.nvd, _2_MockFixedIntBlock_0.tii, _2_Asserting_0.tip, _2.si, _0_Asserting_0.tip, _0_MockFixedIntBlock_0.doc, _0_Direct_0.pos, _2_Lucene41_0.pos, _0_MockFixedIntBlock_0.frq, _2_MockFixedIntBlock_0.doc, _2_Asserting_0.doc, _2_Asserting_0.pos, _0_MockFixedIntBlock_0.pyl, _2_MockFixedIntBlock_0.skp, _2_Asserting_0.tim, _2_Lucene41_0.tim, _2_MockFixedIntBlock_0.pyl, _0.fdt, _2_Lucene41_0.doc, _0.nvd, segments_4, _0.fdx, _0_Asserting_0.doc]
[junit4:junit4]   2> 15886 T141 C7 P58139 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[_0_Lucene41_0.pos, _2_Direct_0.doc, _2_MockFixedIntBlock_0.tib, _2.fdt, _0_MockFixedIntBlock_0.tib, _2.fnm, _0_1.del, _0_Direct_0.tim, _2.nvm, _2_MockFixedIntBlock_0.frq, _0.si, _0_Lucene41_0.tim, _2_Lucene41_0.tip, _0_Direct_0.tip, _0.fnm, _0_Lucene41_0.doc, _2_Direct_0.tip, _0_Lucene41_0.tip, _2.fdx, _0_Asserting_0.pos, _2_Direct_0.tim, _0.nvm, _2_Direct_0.pos, _0_Direct_0.doc, _2_MockFixedIntBlock_0.pos, _0_MockFixedIntBlock_0.pos, _0_Asserting_0.tim, _0_MockFixedIntBlock_0.tii, _0_MockFixedIntBlock_0.skp, _2.nvd, _2_MockFixedIntBlock_0.tii, _2_Asserting_0.tip, _2.si, _0_Asserting_0.tip, _0_MockFixedIntBlock_0.doc, _0_Direct_0.pos, _2_Lucene41_0.pos, _0_MockFixedIntBlock_0.frq, _2_MockFixedIntBlock_0.doc, _2_Asserting_0.doc, _2_Asserting_0.pos, _0_MockFixedIntBlock_0.pyl, _2_MockFixedIntBlock_0.skp, _2_Asserting_0.tim, _2_Lucene41_0.tim, _2_MockFixedIntBlock_0.pyl, _0.fdt, _2_Lucene41_0.doc, _0.nvd, segments_4, _0.fdx, _0_Asserting_0.doc]
[junit4:junit4]   2> 15888 T193 C7 P58139 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 15888 T141 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 15889 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[56 (-1429684066683191296)]} 0 21
[junit4:junit4]   2> 15889 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10059 (1429684066684239872)]} 0 19
[junit4:junit4]   2> 15890 T146 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d742119880008
[junit4:junit4]   2> 15890 T163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 15891 T141 oaz.ZooKeeper.close Session: 0x13d742119880008 closed
[junit4:junit4]   2> 15891 T143 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57286 which had sessionid 0x13d742119880008
[junit4:junit4]   2> 15892 T170 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 15892 T183 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 15892 T170 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 15892 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 15893 T163 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 15893 T159 C8 P45612 /update {version=2&wt=javabin} {add=[59 (1429684066707308544)]} 0 1
[junit4:junit4]   2> 15897 T179 C9 P42845 /update {version=2&wt=javabin} {add=[59 (1429684066711502848)]} 0 1
[junit4:junit4]   2> 15913 T141 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4]   2> 15943 T157 C8 P45612 /update {version=2&wt=javabin} {add=[10060 (1429684066758688768)]} 0 3
[junit4:junit4]   2> 15949 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10060 (1429684066764980224)]} 0 2
[junit4:junit4]   2> 15950 T158 C8 P45612 /update {version=2&wt=javabin} {add=[60 (1429684066766028800)]} 0 3
[junit4:junit4]   2> 15956 T176 C9 P42845 /update {version=2&wt=javabin} {add=[60 (1429684066772320256)]} 0 2
[junit4:junit4]   2> 16044 T159 C8 P45612 /update {version=2&wt=javabin} {delete=[10057 (-1429684066866692096)]} 0 0
[junit4:junit4]   2> 16047 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[10057 (-1429684066869837824)]} 0 0
[junit4:junit4]   2> 16050 T157 C8 P45612 /update {version=2&wt=javabin} {delete=[57 (-1429684066872983552)]} 0 0
[junit4:junit4]   2> 16051 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10061 (1429684066872983553)]} 0 1
[junit4:junit4]   2> 16053 T179 C9 P42845 /update {version=2&wt=javabin} {delete=[57 (-1429684066875080704)]} 0 1
[junit4:junit4]   2> 16054 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10061 (1429684066876129280)]} 0 1
[junit4:junit4]   2> 16056 T158 C8 P45612 /update {version=2&wt=javabin} {add=[61 (1429684066878226432)]} 0 1
[junit4:junit4]   2> 16060 T177 C9 P42845 /update {version=2&wt=javabin} {add=[61 (1429684066882420736)]} 0 2
[junit4:junit4]   2> 16063 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10062 (1429684066885566464)]} 0 1
[junit4:junit4]   2> 16068 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10062 (1429684066890809344)]} 0 2
[junit4:junit4]   2> 16069 T157 C8 P45612 /update {version=2&wt=javabin} {add=[62 (1429684066891857920)]} 0 2
[junit4:junit4]   2> 16073 T179 C9 P42845 /update {version=2&wt=javabin} {add=[62 (1429684066896052224)]} 0 1
[junit4:junit4]   2> 16081 T156 C8 P45612 /update {version=2&wt=javabin} {delete=[10058 (-1429684066905489408)]} 0 0
[junit4:junit4]   2> 16084 T178 C9 P42845 /update {version=2&wt=javabin} {delete=[10058 (-1429684066908635136)]} 0 0
[junit4:junit4]   2> 16086 T158 C8 P45612 /update {version=2&wt=javabin} {delete=[58 (-1429684066910732288)]} 0 0
[junit4:junit4]   2> 16088 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10063 (1429684066911780864)]} 0 2
[junit4:junit4]   2> 16089 T177 C9 P42845 /update {version=2&wt=javabin} {delete=[58 (-1429684066913878016)]} 0 0
[junit4:junit4]   2> 16091 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10063 (1429684066914926592)]} 0 1
[junit4:junit4]   2> 16093 T157 C8 P45612 /update {version=2&wt=javabin} {add=[63 (1429684066917023744)]} 0 1
[junit4:junit4]   2> 16096 T179 C9 P42845 /update {version=2&wt=javabin} {add=[63 (1429684066920169472)]} 0 1
[junit4:junit4]   2> 16153 T156 C8 P45612 /update {version=2&wt=javabin} {add=[10064 (1429684066978889728)]} 0 3
[junit4:junit4]   2> 16158 T158 C8 P45612 /update {version=2&wt=javabin} {add=[64 (1429684066983084032)]} 0 3
[junit4:junit4]   2> 16159 T178 C9 P42845 /update {version=2&wt=javabin} {add=[10064 (1429684066986229760)]} 0 2
[junit4:junit4]   2> 16163 T177 C9 P42845 /update {version=2&wt=javabin} {add=[64 (1429684066990424064)]} 0 2
[junit4:junit4]   2> 16167 T141 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 16170 T141 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58139
[junit4:junit4]   2> 16170 T141 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 16171 T141 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 16171 T141 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237
[junit4:junit4]   2> 16171 T141 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/solr.xml
[junit4:junit4]   2> 16172 T141 oasc.CoreContainer.<init> New CoreContainer 609850803
[junit4:junit4]   2> 16172 T141 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/'
[junit4:junit4]   2> 16172 T141 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363452967237/'
[junit4:junit4]   2> 16179 T159 C8 P45612 /update {version=2&wt=javabin} {add=[10065 (1429684067007201280)]} 0 1
[junit4:junit4]   2> 16183 T176 C9 P42845 /update {version=2&wt=javabin} {add=[10065 (1429684067011395584)]} 0 1
[junit4:junit4]   2> 16184 T157 C8 P45612 /update {version=2&wt=javabin} {add=[65 (1429684067012444160)]} 0 1
[junit4:junit4]   2> 16188 T179 C9 P42845 /update {version=2&wt=javabin} {add=[65 (1429684067016638464)]} 0 1
[junit4:junit4]   2> 16195 T141 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 16195 T141 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 16195 T141 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 16196 T141 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 16196 T141 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 16196 T141 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 16197 T1

[...truncated too long message...]

nit4]   1>                   "core":"unloadcollection2"}}}},
[junit4:junit4]   1>           "router":"compositeId"}}
[junit4:junit4]   1>   /solr/live_nodes (5)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:44544_qp_qtp%2Fgm (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:48307_qp_qtp%2Fgm (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:53930_qp_qtp%2Fgm (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:33407_qp_qtp%2Fgm (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:51245_qp_qtp%2Fgm (0)
[junit4:junit4]   1>  /zookeeper (1)
[junit4:junit4]   1>  DATA:
[junit4:junit4]   1>      
[junit4:junit4]   1> 
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=UnloadDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=DDA8E93F6FE5B250 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ru_RU -Dtests.timezone=America/Menominee -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE  138s J0 | UnloadDistributedZkTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: Still found shard
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([DDA8E93F6FE5B250:5C4E672718BAD26C]:0)
[junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > 	at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:124)
[junit4:junit4]    > 	at org.apache.solr.cloud.UnloadDistributedZkTest.doTest(UnloadDistributedZkTest.java:77)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:487)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 138074 T1329 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 138088 T1328 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4]   2> 138724 T1332 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 139181 T1411 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 139181 T1411 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 139181 T1411 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {intDefault=PostingsFormat(name=MockRandom), other_tl1=PostingsFormat(name=MockRandom), range_facet_si=PostingsFormat(name=MockSep), _version_=PostingsFormat(name=MockRandom), range_facet_l=PostingsFormat(name=MockSep), multiDefault=PostingsFormat(name=MockSep), timestamp=PostingsFormat(name=MockSep), id=Pulsing41(freqCutoff=7 minBlockSize=82 maxBlockSize=245), a_si=PostingsFormat(name=MockRandom), text=MockFixedIntBlock(blockSize=1941), a_t=PostingsFormat(name=MockSep), range_facet_sl=Pulsing41(freqCutoff=7 minBlockSize=82 maxBlockSize=245)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ru_RU, timezone=America/Menominee
[junit4:junit4]   2> NOTE: Linux 3.2.0-38-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=1,free=143096088,total=278724608
[junit4:junit4]   2> NOTE: All tests run in this JVM: [SpatialFilterTest, SoftAutoCommitTest, DateFieldTest, ZkControllerTest, TestRemoteStreaming, PeerSyncTest, RecoveryZkTest, ExternalFileFieldSortTest, SimpleFacetsTest, TestCharFilters, LegacyHTMLStripCharFilterTest, QueryEqualityTest, TestFuzzyAnalyzedSuggestions, DocumentBuilderTest, TestRandomDVFaceting, ChaosMonkeyNothingIsSafeTest, TestSolrQueryParser, ClusterStateUpdateTest, TestStressVersions, TestConfig, AliasIntegrationTest, TestShardHandlerFactory, TestTrie, DistributedQueryElevationComponentTest, TestPropInjectDefaults, TestReplicationHandler, QueryElevationComponentTest, QueryParsingTest, TestPseudoReturnFields, DirectSolrConnectionTest, TestBadConfig, TestFaceting, SolrCoreTest, UUIDFieldTest, UnloadDistributedZkTest]
[junit4:junit4] Completed on J0 in 139.38s, 1 test, 1 failure <<< FAILURES!

[...truncated 551 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:381: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:361: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1213: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:877: There were test failures: 273 suites, 1147 tests, 2 suite-level errors, 1 error, 1 failure, 13 ignored (7 assumptions)

Total time: 39 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 64bit/jdk1.8.0-ea-b79 -XX:+UseParallelGC
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message