lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_51) - Build # 9344 - Still Failing!
Date Wed, 12 Feb 2014 13:11:39 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9344/
Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=347 sum(shards)=348 cloudClient=348

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=347 sum(shards)=348 cloudClient=348
	at __randomizedtesting.SeedInfo.seed([A1DEEDABE4CE5891:203863B3939138AD]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1215)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:208)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:


Stack Trace:
org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([A1DEEDABE4CE5891:A5D66258F66BB7B0]:0)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/91239849777692675-node1_core1-n_
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:314)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:243)
	... 42 more




Build Log:
[...truncated 10347 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 295975 T1356 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1392209595890
   [junit4]   2> 295975 T1356 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 295991 T1356 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
   [junit4]   2> 295992 T1356 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 295997 T1357 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 296096 T1356 oasc.ZkTestServer.run start zk server on port:48853
   [junit4]   2> 296097 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296174 T1363 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d759072 name:ZooKeeperConnection Watcher:127.0.0.1:48853/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296175 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296176 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296179 T1365 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21a70246 name:ZooKeeperConnection Watcher:127.0.0.1:48853 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296179 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296181 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296183 T1367 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b6e9754 name:ZooKeeperConnection Watcher:127.0.0.1:48853 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296183 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296183 T1356 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 296188 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 296191 T1356 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 296194 T1356 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 296196 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296199 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296201 T1369 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ee304b name:ZooKeeperConnection Watcher:127.0.0.1:48853/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296201 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296202 T1356 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 296204 T1356 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 296209 T1356 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:48853_solr
   [junit4]   2> 296210 T1356 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 296213 T1356 oasc.Overseer.start Overseer (id=91239848078016515-127.0.0.1:48853_solr-n_0000000000) starting
   [junit4]   2> 296216 T1356 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 296221 T1356 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 296229 T1356 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 296240 T1371 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 296242 T1371 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 296243 T1356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 296243 T1370 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 296248 T1370 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 296249 T1370 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 296249 T1370 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 296253 T1369 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296254 T1363 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 296297 T1356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 296299 T1369 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296301 T1370 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 296302 T1370 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 296302 T1370 oasc.Overseer$ClusterStateUpdater.updateState node=core_node1 is already registered
   [junit4]   2> 296302 T1370 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 296305 T1369 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 296407 T1363 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 296454 T1356 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:48853 48853
   [junit4]   2> 296595 T1356 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 296601 T1356 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 296601 T1356 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 296602 T1372 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 296702 T1356 oasc.ZkTestServer.run start zk server on port:35793
   [junit4]   2> 296703 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296707 T1378 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d129ae1 name:ZooKeeperConnection Watcher:127.0.0.1:35793 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296708 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296727 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296730 T1380 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6262e75f name:ZooKeeperConnection Watcher:127.0.0.1:35793 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296730 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296731 T1356 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 296734 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296736 T1382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d096756 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296737 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296737 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 296739 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296741 T1384 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57871d53 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296742 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296742 T1356 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 296746 T1356 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 296750 T1356 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:35793_solr
   [junit4]   2> 296751 T1356 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 296753 T1356 oasc.Overseer.start Overseer (id=91239848117796867-127.0.0.1:35793_solr-n_0000000000) starting
   [junit4]   2> 296758 T1356 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 296761 T1356 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 296765 T1356 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 296767 T1386 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 296769 T1385 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 296769 T1386 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 296770 T1356 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 296778 T1356 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 296780 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296785 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296787 T1388 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@309a0490 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296788 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296789 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296792 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 296797 T1382 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 296797 T1388 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 296798 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296800 T1390 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@593f8179 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296800 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296801 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296804 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 296808 T1382 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 296809 T1388 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 296809 T1390 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 296809 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296813 T1392 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b65b786 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296813 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296815 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296820 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 296825 T1382 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 296826 T1390 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 296827 T1388 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 296827 T1392 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 296828 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296829 T1394 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b1b41ad name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296830 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296832 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296834 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 296837 T1390 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 296837 T1382 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 296838 T1388 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 296838 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296838 T1392 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 296839 T1394 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 296840 T1396 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49b87cd3 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296840 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296842 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296844 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 296847 T1390 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 296847 T1392 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 296848 T1388 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 296848 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296848 T1394 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 296848 T1382 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 296848 T1396 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 296850 T1398 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d947b7c name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296850 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296851 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296855 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 296860 T1394 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 296860 T1392 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 296860 T1396 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 296860 T1388 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 296861 T1390 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 296861 T1382 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 296861 T1398 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 296861 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296863 T1400 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@274a64b6 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296863 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296865 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296868 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 296873 T1394 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 296873 T1390 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 296873 T1388 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 296873 T1392 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 296874 T1382 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 296874 T1396 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 296873 T1398 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 296873 T1400 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 296876 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296877 T1402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f5ee039 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296877 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296878 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296881 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 296883 T1400 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 296884 T1388 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 296884 T1392 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 296884 T1382 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 296884 T1396 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 296883 T1394 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 296884 T1398 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 296884 T1390 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 296886 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296888 T1402 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 296888 T1404 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e12f4c1 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296889 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296891 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296894 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 296897 T1398 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296897 T1396 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296898 T1382 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296897 T1388 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296897 T1390 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296898 T1392 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296898 T1400 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296898 T1394 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296899 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296899 T1404 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296901 T1406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c90e352 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296901 T1402 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 296901 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296902 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296906 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 296912 T1396 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296912 T1402 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296912 T1406 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296912 T1392 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296912 T1400 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296913 T1382 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296913 T1394 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296912 T1398 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296912 T1388 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296913 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296913 T1390 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296913 T1404 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 296915 T1408 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ad14d1d name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296915 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296916 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296919 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 296923 T1394 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296924 T1396 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296925 T1388 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296926 T1392 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296924 T1404 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296924 T1402 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296924 T1400 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296926 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296926 T1408 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296926 T1406 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296926 T1382 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296925 T1390 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296925 T1398 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 296928 T1410 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@512a8911 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296928 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296930 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296935 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 296941 T1404 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296941 T1406 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296942 T1408 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296942 T1394 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296942 T1402 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296942 T1390 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296942 T1392 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296943 T1396 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296943 T1382 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296943 T1388 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296943 T1400 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296943 T1398 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296944 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296944 T1410 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 296946 T1412 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1dfc6021 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296947 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296949 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296953 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 296956 T1396 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296956 T1390 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296957 T1382 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296957 T1398 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296957 T1408 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296958 T1402 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296958 T1406 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296958 T1400 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296958 T1392 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296959 T1394 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296959 T1410 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296960 T1388 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296960 T1404 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296962 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296964 T1412 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 296965 T1414 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a63ef51 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296967 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296970 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 296981 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 296984 T1402 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296984 T1394 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296985 T1392 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296986 T1388 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296986 T1406 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296987 T1408 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296986 T1398 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296987 T1396 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296987 T1382 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296987 T1404 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296987 T1390 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296988 T1400 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296988 T1414 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296988 T1410 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296989 T1412 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 296995 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296998 T1416 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d34ea4 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296998 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297003 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297007 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 297015 T1412 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297015 T1408 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297015 T1402 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297015 T1382 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297016 T1400 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297016 T1398 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297016 T1406 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297015 T1410 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297016 T1390 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297016 T1396 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297021 T1418 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@453cc9a2 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297020 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297019 T1416 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297019 T1414 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297017 T1394 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297017 T1392 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297024 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297017 T1388 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297017 T1404 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 297031 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297033 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 297037 T1404 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297037 T1398 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297038 T1412 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297038 T1416 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297038 T1392 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297038 T1400 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297038 T1388 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297039 T1402 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297039 T1406 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297039 T1410 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297039 T1394 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297039 T1390 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297039 T1382 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297039 T1408 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297039 T1396 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297039 T1414 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297041 T1418 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 297044 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297045 T1420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30ce57c5 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297045 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297046 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297049 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 297052 T1382 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1406 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1412 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1388 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1410 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1396 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1414 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297054 T1408 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1394 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297052 T1400 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1392 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1402 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1398 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297053 T1390 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297058 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297058 T1418 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297059 T1404 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297059 T1420 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297060 T1422 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5146b037 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297060 T1416 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 297060 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297063 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297065 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 297071 T1412 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297071 T1414 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297071 T1382 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297072 T1416 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297072 T1404 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297072 T1388 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297072 T1418 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297072 T1390 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297073 T1408 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297073 T1410 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297074 T1400 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297074 T1394 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297074 T1396 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297074 T1402 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297075 T1420 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297075 T1398 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297075 T1406 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297076 T1422 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297076 T1392 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 297077 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297079 T1424 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f33d722 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297079 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297080 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297083 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 297086 T1382 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297086 T1408 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297086 T1390 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297086 T1416 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297087 T1422 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297086 T1396 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297087 T1412 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297087 T1400 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297088 T1392 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297088 T1402 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297088 T1406 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297089 T1414 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297089 T1404 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297089 T1420 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297089 T1388 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297089 T1398 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297090 T1410 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297090 T1418 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297090 T1394 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297092 T1424 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 297092 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297094 T1426 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72a48fe4 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297095 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297096 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297100 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 297110 T1392 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297110 T1398 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297110 T1414 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297111 T1394 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297111 T1396 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297112 T1400 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297112 T1406 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297112 T1382 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297112 T1404 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297112 T1412 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297113 T1422 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297113 T1402 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297112 T1410 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297113 T1420 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297113 T1390 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297113 T1388 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297113 T1416 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297113 T1418 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297114 T1426 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297114 T1424 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297114 T1408 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 297116 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297119 T1428 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ac86066 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297119 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297123 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297127 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 297140 T1392 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297141 T1426 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297141 T1406 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297141 T1394 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297141 T1398 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297142 T1396 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297142 T1388 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297143 T1402 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297142 T1400 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297143 T1424 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297143 T1420 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297143 T1390 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297144 T1382 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297144 T1414 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297144 T1408 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297144 T1416 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297144 T1412 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297143 T1418 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297145 T1422 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297144 T1404 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297144 T1410 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297146 T1428 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 297147 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297151 T1430 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@465faac2 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297152 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297153 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297156 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 297164 T1408 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297165 T1390 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297165 T1420 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297165 T1410 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297165 T1418 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297165 T1388 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297166 T1422 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297166 T1416 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297166 T1382 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297166 T1404 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297166 T1412 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297166 T1414 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297167 T1406 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297167 T1398 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297167 T1392 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297167 T1430 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297167 T1428 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297168 T1396 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297168 T1424 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297169 T1400 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297168 T1402 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297169 T1394 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297169 T1426 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 297170 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297173 T1432 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66fe3348 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297174 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297175 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297176 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 297181 T1416 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297181 T1404 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297182 T1414 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297182 T1422 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297182 T1408 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297182 T1382 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297183 T1410 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297183 T1412 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297183 T1420 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297184 T1432 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297184 T1418 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297184 T1402 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297185 T1394 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297185 T1388 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297186 T1390 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297187 T1396 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297187 T1400 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297187 T1424 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297188 T1406 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297188 T1392 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297188 T1426 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297188 T1398 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297189 T1430 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297189 T1428 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 297190 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297192 T1434 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@795c8175 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297192 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297194 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297198 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 297203 T1412 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297204 T1430 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297203 T1428 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297204 T1414 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297203 T1382 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297203 T1416 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297204 T1388 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297204 T1422 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297205 T1426 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297205 T1400 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297206 T1410 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297206 T1398 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297206 T1418 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297206 T1394 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297206 T1432 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297207 T1408 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297207 T1402 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297207 T1396 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297206 T1424 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297207 T1392 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297207 T1390 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297208 T1420 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297208 T1406 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297208 T1404 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297209 T1434 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 297210 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297217 T1436 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@127208c7 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297218 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297220 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297226 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 297232 T1416 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297233 T1402 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297233 T1396 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297233 T1414 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297233 T1398 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297233 T1410 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297234 T1426 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297234 T1424 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297234 T1436 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297234 T1408 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297234 T1432 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297235 T1404 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297235 T1420 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297235 T1390 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297235 T1388 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297235 T1412 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297236 T1394 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297235 T1392 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297236 T1418 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297236 T1406 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297236 T1422 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297235 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297237 T1428 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297237 T1400 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297237 T1382 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297236 T1430 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 297238 T1438 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fec499f name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297238 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297239 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297243 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 297251 T1428 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297251 T1404 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297251 T1416 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297251 T1422 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297251 T1410 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297251 T1382 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297252 T1432 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297252 T1388 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297252 T1418 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297252 T1436 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297253 T1426 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297253 T1402 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297253 T1408 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297254 T1396 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297254 T1412 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297254 T1414 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297254 T1406 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297254 T1420 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297255 T1400 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297255 T1390 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297255 T1424 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297255 T1394 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297256 T1392 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297256 T1430 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297256 T1434 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297255 T1398 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297258 T1438 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 297266 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297266 T1440 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eea3542 name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297267 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297268 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297272 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 297275 T1408 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297275 T1428 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297276 T1404 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297276 T1406 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297276 T1418 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297276 T1388 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297277 T1436 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297278 T1430 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297278 T1422 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297278 T1426 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297279 T1412 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297278 T1432 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297279 T1394 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297279 T1390 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297278 T1382 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297280 T1398 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297279 T1402 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297280 T1434 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297280 T1420 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297280 T1400 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297280 T1438 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297279 T1424 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297279 T1416 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297279 T1396 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297281 T1440 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297279 T1410 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297280 T1414 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297280 T1392 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 297281 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297284 T1442 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49a4835a name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297285 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297286 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297289 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 297294 T1426 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297294 T1418 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297294 T1406 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297297 T1382 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297297 T1442 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297297 T1388 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297297 T1420 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297298 T1394 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297299 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297300 T1404 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297300 T1396 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297300 T1438 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297300 T1424 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297300 T1400 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297301 T1414 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297300 T1402 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297300 T1416 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297301 T1422 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297301 T1440 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297302 T1444 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4447e86f name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297302 T1412 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297301 T1390 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297301 T1436 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297303 T1432 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297303 T1398 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297301 T1434 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297301 T1430 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297303 T1408 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297302 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297302 T1392 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297302 T1428 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297303 T1410 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 297305 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297307 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
   [junit4]   2> 297316 T1406 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297316 T1402 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297316 T1444 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297316 T1426 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297317 T1424 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297317 T1396 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297318 T1408 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297318 T1388 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297318 T1410 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297318 T1432 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297318 T1418 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297319 T1436 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297319 T1404 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297319 T1428 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297319 T1434 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297319 T1382 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297320 T1398 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297320 T1422 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297320 T1392 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297320 T1416 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297320 T1430 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297320 T1394 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297321 T1400 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297321 T1420 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297321 T1390 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297321 T1440 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297321 T1442 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297321 T1438 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297321 T1414 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297322 T1412 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 297322 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297324 T1446 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d958a8c name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297324 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297325 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297328 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node29
   [junit4]   2> 297335 T1406 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297335 T1446 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297336 T1444 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297336 T1402 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297336 T1426 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297337 T1396 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297339 T1424 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297339 T1432 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297340 T1410 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297340 T1408 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297340 T1436 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297340 T1434 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297341 T1388 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297341 T1418 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297341 T1442 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297342 T1416 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297342 T1404 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297342 T1430 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297343 T1398 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297343 T1438 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297343 T1394 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297343 T1392 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297344 T1390 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297344 T1400 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297344 T1440 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297344 T1382 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297344 T1420 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297344 T1422 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297345 T1428 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297344 T1414 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297345 T1412 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 297346 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297347 T1448 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31d656dc name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297348 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297349 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297352 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node30
   [junit4]   2> 297370 T1432 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297370 T1410 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297370 T1442 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297371 T1408 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297371 T1436 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297371 T1434 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297378 T1404 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297379 T1396 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297379 T1428 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297380 T1440 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297380 T1444 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297380 T1406 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297380 T1430 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297381 T1394 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297381 T1390 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297381 T1416 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297382 T1420 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297382 T1426 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297382 T1414 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297382 T1446 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297383 T1412 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297383 T1424 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297383 T1392 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297383 T1402 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297383 T1382 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297384 T1400 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297384 T1448 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297384 T1398 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297384 T1422 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297384 T1418 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297385 T1388 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297385 T1438 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 297391 T1356 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297393 T1450 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@763b00bc name:ZooKeeperConnection Watcher:127.0.0.1:35793/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297393 T1356 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297395 T1356 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297397 T1356 oascc.SolrZkClient.makePath makePath: /live_nodes/node31
   [junit4]   2> 297407 T1414 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297407 T1440 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297408 T1412 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297408 T1420 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297408 T1390 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297409 T1394 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297409 T1400 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297409 T1438 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297410 T1398 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297410 T1430 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 297411 T1392 oascc.ZkStateReader$3.process Updating live nodes..

[...truncated too long message...]

othingIsSafeTest-1392209815555/jetty6/index
   [junit4]   2> 596904 T2330 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392209815555/jetty6 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392209815555/jetty6;done=false>>]
   [junit4]   2> 596905 T2330 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392209815555/jetty6
   [junit4]   2> 596905 T2330 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 596906 T2330 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 596973 T2350 C5944 P42566 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 596974 T2350 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 596975 T2350 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@655100a1
   [junit4]   2> 596977 T2350 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=311,cumulative_deletesById=146,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=1}
   [junit4]   2> 596977 T2350 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 596978 T2350 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 596978 T2350 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 596979 T2350 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 596980 T2350 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 596980 T2350 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392209815555/jetty7 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392209815555/jetty7;done=false>>]
   [junit4]   2> 596981 T2350 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392209815555/jetty7
   [junit4]   2> 596981 T2350 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392209815555/jetty7/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392209815555/jetty7/index;done=false>>]
   [junit4]   2> 596981 T2350 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1392209815555/jetty7/index
   [junit4]   2> 596981 T2350 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 596982 T2350 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene3x, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=ar_AE, timezone=MST
   [junit4]   2> NOTE: Linux 3.8.0-35-generic amd64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=8,threads=1,free=29437984,total=300187648
   [junit4]   2> NOTE: All tests run in this JVM: [ConvertedLegacyTest, TestBinaryResponseWriter, PrimUtilsTest, TimeZoneUtilsTest, BadIndexSchemaTest, TestSchemaResource, TestMaxScoreQueryParser, TestReloadAndDeleteDocs, TestCursorMarkWithoutUniqueKey, CursorPagingTest, TermVectorComponentDistributedTest, SimplePostToolTest, SpellCheckCollatorTest, TestPartialUpdateDeduplication, OutputWriterTest, BasicDistributedZk2Test, TestUpdate, SolrPluginUtilsTest, XmlUpdateRequestHandlerTest, TestIndexingPerformance, ChangedSchemaMergeTest, TestSolrXmlPersistence, TestStressReorder, TestSolrCoreProperties, TestSolrXml, DistributedSuggestComponentTest, TestDistributedMissingSort, NoCacheHeaderTest, TestFoldingMultitermQuery, RequestHandlersTest, CoreContainerCoreInitFailuresTest, TestStressRecovery, ZkSolrClientTest, DocValuesMultiTest, PreAnalyzedFieldTest, TestCollationKeyRangeQueries, DocumentAnalysisRequestHandlerTest, TestSchemaNameResource, XsltUpdateRequestHandlerTest, TestOmitPositions, TestImplicitCoreProperties, AssignTest, HighlighterConfigTest, TestRecoveryHdfs, TestAtomicUpdateErrorCases, TestUtils, JSONWriterTest, MinimalSchemaTest, TestMultiCoreConfBootstrap, DistanceFunctionTest, TestSolrQueryParserDefaultOperatorResource, OpenCloseCoreStressTest, TestSimpleQParserPlugin, TestFastWriter, StandardRequestHandlerTest, PingRequestHandlerTest, TestZkChroot, TestFiltering, TestSweetSpotSimilarityFactory, TestCollationField, HdfsLockFactoryTest, HardAutoCommitTest, TestFastLRUCache, TestSolrJ, PreAnalyzedUpdateProcessorTest, TestCoreDiscovery, SuggesterWFSTTest, TestCSVLoader, SolrIndexConfigTest, TestRTGBase, SortByFunctionTest, TestBinaryField, TestDistribDocBasedVersion, MultiTermTest, ShardRoutingCustomTest, RequiredFieldsTest, TestArbitraryIndexDir, TestFreeTextSuggestions, TestDistributedSearch, ChaosMonkeySafeLeaderTest, StressHdfsTest, UniqFieldsUpdateProcessorFactoryTest, DisMaxRequestHandlerTest, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed on J1 in 82.34s, 1 test, 1 failure <<< FAILURES!

[...truncated 599 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1308: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:941: There were test failures: 362 suites, 1562 tests, 1 error, 1 failure, 36 ignored (8 assumptions)

Total time: 44 minutes 41 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_51 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message