lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.8-Linux (32bit/jdk1.7.0_60-ea-b14) - Build # 45 - Failure!
Date Tue, 22 Apr 2014 14:06:49 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.8-Linux/45/
Java: 32bit/jdk1.7.0_60-ea-b14 -server -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed

Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
	at __randomizedtesting.SeedInfo.seed([1D2C2210D9BE7786:1924ADE3CB1B98A7]:0)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:136)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:142)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:110)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:134)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:267)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:661)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:40)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:123)
	... 46 more
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:128)
	at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:34)
	... 47 more




Build Log:
[...truncated 11123 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-1D2C2210D9BE7786-001/init-core-data-001
   [junit4]   2> 725211 T4180 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 725212 T4180 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 725212 T4180 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 725214 T4180 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
   [junit4]   2> 725215 T4180 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 725215 T4181 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 725315 T4180 oasc.ZkTestServer.run start zk server on port:40590
   [junit4]   2> 725316 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730621 T4184 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 5303ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
   [junit4]   2> 730623 T4187 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@634821 name:ZooKeeperConnection Watcher:127.0.0.1:40590 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730624 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730627 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730629 T4189 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@824e1 name:ZooKeeperConnection Watcher:127.0.0.1:40590 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730629 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730630 T4180 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 730635 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730637 T4191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9d4907 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730638 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730638 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 730640 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730642 T4193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17b5c22 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730642 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730643 T4180 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 730646 T4180 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 730649 T4180 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40590_solr
   [junit4]   2> 730649 T4180 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 730652 T4180 oasc.Overseer.start Overseer (id=91630792381693955-127.0.0.1:40590_solr-n_0000000000) starting
   [junit4]   2> 730654 T4180 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 730658 T4180 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 730660 T4180 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 730662 T4180 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 730665 T4180 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 730667 T4180 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 730672 T4195 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 730672 T4180 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 730673 T4195 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:40590_solr
   [junit4]   2> 730673 T4194 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 730675 T4180 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 730676 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730678 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730679 T4197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8a0ce0 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730679 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730680 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730681 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
   [junit4]   2> 730683 T4197 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 730683 T4191 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 730683 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730684 T4199 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23aec2 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730684 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730685 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730686 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 730688 T4191 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 730688 T4197 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 730688 T4199 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 730688 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730689 T4201 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34370 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730689 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730690 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730691 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 730693 T4191 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 730693 T4199 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 730693 T4201 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 730693 T4197 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 730694 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730695 T4203 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ba7444 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730695 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730696 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730697 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
   [junit4]   2> 730699 T4199 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 730699 T4197 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 730699 T4203 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 730699 T4201 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 730699 T4191 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 730700 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730701 T4205 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1aeb17 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730701 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730702 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730704 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
   [junit4]   2> 730705 T4199 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 730705 T4201 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 730705 T4197 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 730706 T4191 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 730706 T4203 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 730706 T4205 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 730706 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730707 T4207 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17afde0 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730707 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730708 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730709 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
   [junit4]   2> 730711 T4199 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 730711 T4201 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 730711 T4203 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 730711 T4197 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 730711 T4207 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 730711 T4205 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 730711 T4191 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 730712 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730713 T4209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@182c451 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730713 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730713 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730716 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
   [junit4]   2> 730718 T4201 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 730718 T4191 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 730718 T4199 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 730719 T4209 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 730718 T4207 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 730718 T4205 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 730719 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730718 T4203 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 730718 T4197 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 730720 T4211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cf0082 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730720 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730721 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730723 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
   [junit4]   2> 730724 T4199 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 730724 T4191 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 730724 T4207 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 730725 T4205 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 730725 T4211 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 730725 T4209 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 730725 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730724 T4203 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 730724 T4201 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 730725 T4197 oascc.ZkStateReader$3.process Updating live nodes... (8)
   [junit4]   2> 730726 T4213 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1471948 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730726 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730727 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730728 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
   [junit4]   2> 730730 T4199 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730730 T4191 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730730 T4201 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730730 T4205 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730730 T4209 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730730 T4213 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730730 T4211 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730730 T4203 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730730 T4197 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730730 T4207 oascc.ZkStateReader$3.process Updating live nodes... (9)
   [junit4]   2> 730731 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730732 T4215 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e94df name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730732 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730734 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730736 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node9
   [junit4]   2> 730739 T4201 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730739 T4209 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730739 T4205 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730739 T4197 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730739 T4203 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730740 T4215 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730739 T4199 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730740 T4211 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730739 T4191 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730739 T4213 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730739 T4207 oascc.ZkStateReader$3.process Updating live nodes... (10)
   [junit4]   2> 730740 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730741 T4217 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14c4947 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730741 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730742 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730744 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node10
   [junit4]   2> 730746 T4199 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730747 T4213 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730746 T4217 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730747 T4209 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730747 T4201 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730747 T4191 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730747 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730747 T4211 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730747 T4207 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730747 T4203 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730747 T4215 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730747 T4205 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730749 T4219 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a7eb44 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730747 T4197 oascc.ZkStateReader$3.process Updating live nodes... (11)
   [junit4]   2> 730749 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730749 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730751 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node11
   [junit4]   2> 730752 T4205 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730753 T4213 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730753 T4201 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730753 T4209 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730753 T4191 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730753 T4207 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730753 T4199 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730753 T4217 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730753 T4203 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730752 T4211 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730752 T4197 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730752 T4215 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730754 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730755 T4221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13ee40c name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730754 T4219 oascc.ZkStateReader$3.process Updating live nodes... (12)
   [junit4]   2> 730756 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730756 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730758 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node12
   [junit4]   2> 730760 T4213 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4207 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4217 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4211 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4203 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4199 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4205 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4209 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4219 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4197 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730760 T4221 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730762 T4223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15f4939 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730761 T4191 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730761 T4201 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730761 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730761 T4215 oascc.ZkStateReader$3.process Updating live nodes... (13)
   [junit4]   2> 730766 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730767 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730768 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node13
   [junit4]   2> 730770 T4213 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730770 T4209 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4205 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4217 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4223 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4219 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4197 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4191 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4203 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4207 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4199 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4221 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4215 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730771 T4211 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730772 T4225 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f6df9 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730771 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730771 T4201 oascc.ZkStateReader$3.process Updating live nodes... (14)
   [junit4]   2> 730773 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730775 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730776 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node14
   [junit4]   2> 730778 T4207 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730778 T4197 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730778 T4203 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730779 T4215 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730779 T4221 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730779 T4199 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730779 T4223 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730778 T4213 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730778 T4209 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730778 T4211 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730780 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730779 T4225 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730779 T4217 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730779 T4205 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730779 T4201 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730779 T4191 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730779 T4219 oascc.ZkStateReader$3.process Updating live nodes... (15)
   [junit4]   2> 730781 T4227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1345989 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730781 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730782 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730784 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node15
   [junit4]   2> 730785 T4221 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730786 T4223 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730786 T4215 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730786 T4209 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730786 T4201 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730786 T4197 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730786 T4213 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730787 T4191 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730787 T4227 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730787 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730787 T4203 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730787 T4225 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730787 T4205 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730787 T4219 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730787 T4207 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730787 T4211 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730786 T4199 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730786 T4217 oascc.ZkStateReader$3.process Updating live nodes... (16)
   [junit4]   2> 730788 T4229 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ae787f name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730789 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730789 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730791 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node16
   [junit4]   2> 730793 T4221 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730793 T4219 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730793 T4229 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730794 T4207 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730794 T4227 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730794 T4211 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730793 T4197 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730793 T4225 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730793 T4191 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730793 T4217 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730794 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730794 T4223 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730794 T4203 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730795 T4231 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1410b5f name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730794 T4213 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730794 T4209 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730794 T4201 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730794 T4205 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730794 T4215 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730793 T4199 oascc.ZkStateReader$3.process Updating live nodes... (17)
   [junit4]   2> 730795 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730797 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730798 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node17
   [junit4]   2> 730800 T4221 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730800 T4215 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730800 T4207 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730800 T4213 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4209 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4205 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730800 T4223 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4203 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4211 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4191 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4231 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4199 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4201 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4219 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4225 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4197 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4227 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4217 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730801 T4229 oascc.ZkStateReader$3.process Updating live nodes... (18)
   [junit4]   2> 730802 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730804 T4233 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7180c6 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730804 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730805 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730806 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node18
   [junit4]   2> 730809 T4231 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730809 T4221 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4207 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4211 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4213 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4191 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730809 T4225 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4219 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4197 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730811 T4217 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4199 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4227 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4203 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730811 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730810 T4229 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4209 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4205 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730811 T4223 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730811 T4215 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4201 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730810 T4233 oascc.ZkStateReader$3.process Updating live nodes... (19)
   [junit4]   2> 730813 T4235 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d89d8c name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730813 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730814 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730816 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node19
   [junit4]   2> 730819 T4221 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730819 T4231 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4229 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730819 T4209 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4227 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4211 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4225 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730819 T4215 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730821 T4191 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730821 T4197 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4233 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4203 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4235 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4205 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4199 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4201 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4217 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730822 T4237 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d8fd63 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730820 T4223 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4219 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4207 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730820 T4213 oascc.ZkStateReader$3.process Updating live nodes... (20)
   [junit4]   2> 730821 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730823 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730824 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730825 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node20
   [junit4]   2> 730828 T4231 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730829 T4199 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4213 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4235 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4217 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4225 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730831 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730830 T4203 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4205 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4237 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730829 T4191 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730829 T4215 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730829 T4211 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730829 T4227 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730829 T4197 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730829 T4229 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730828 T4207 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730828 T4209 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730832 T4239 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e7a7b name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730831 T4233 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4221 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4219 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4201 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730830 T4223 oascc.ZkStateReader$3.process Updating live nodes... (21)
   [junit4]   2> 730833 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730835 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730836 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node21
   [junit4]   2> 730839 T4227 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730839 T4201 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730839 T4231 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4197 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4223 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4215 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4191 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4217 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4211 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4219 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730839 T4207 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730839 T4229 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730839 T4235 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730841 T4233 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730841 T4237 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730841 T4209 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730841 T4213 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4221 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4239 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4203 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4199 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4205 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730840 T4225 oascc.ZkStateReader$3.process Updating live nodes... (22)
   [junit4]   2> 730843 T4241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cac056 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730841 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730843 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730844 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730846 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node22
   [junit4]   2> 730848 T4207 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4227 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4237 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4201 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4229 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4233 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4197 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4231 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4221 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730848 T4235 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4203 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4241 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4205 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4215 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4239 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4191 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4225 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4211 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730850 T4199 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4217 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4223 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4213 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4219 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730849 T4209 oascc.ZkStateReader$3.process Updating live nodes... (23)
   [junit4]   2> 730852 T4243 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8001c1 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730851 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730853 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730854 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730856 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node23
   [junit4]   2> 730859 T4199 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730859 T4233 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730859 T4211 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730859 T4197 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4225 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4205 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730859 T4215 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4207 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730859 T4219 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4237 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4231 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4213 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730861 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730860 T4227 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4209 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4221 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4203 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4235 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4239 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4191 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4223 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730859 T4201 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730859 T4217 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730862 T4245 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b23f2b name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730861 T4243 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730861 T4241 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730860 T4229 oascc.ZkStateReader$3.process Updating live nodes... (24)
   [junit4]   2> 730863 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730864 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730866 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node24
   [junit4]   2> 730868 T4239 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730868 T4199 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730868 T4203 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4205 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4191 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4225 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4243 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4217 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4237 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730870 T4231 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730870 T4215 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4209 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4223 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730870 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730870 T4245 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730870 T4229 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4213 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4211 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4197 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4221 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4235 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4227 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4233 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4219 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4201 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4241 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730869 T4207 oascc.ZkStateReader$3.process Updating live nodes... (25)
   [junit4]   2> 730871 T4247 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@df501d name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730873 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730874 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730875 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node25
   [junit4]   2> 730878 T4197 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730878 T4233 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730878 T4205 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730878 T4215 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730878 T4219 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730878 T4241 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4203 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4239 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730878 T4225 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4201 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730880 T4223 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730880 T4227 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730880 T4209 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730880 T4237 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4243 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4229 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4247 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4199 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4191 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4211 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4245 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730879 T4235 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730880 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730880 T4213 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730880 T4207 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730880 T4221 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730880 T4231 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730880 T4217 oascc.ZkStateReader$3.process Updating live nodes... (26)
   [junit4]   2> 730882 T4249 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1991879 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730883 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730883 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730885 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node26
   [junit4]   2> 730887 T4197 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4233 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4211 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4235 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4203 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4215 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4237 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4239 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4191 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4231 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4225 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4209 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4249 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4243 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4201 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4207 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4241 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730889 T4247 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4213 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4205 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4199 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4221 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4219 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4227 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4223 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4245 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4229 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730888 T4217 oascc.ZkStateReader$3.process Updating live nodes... (27)
   [junit4]   2> 730891 T4251 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1999082 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730890 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730892 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730893 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730895 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node27
   [junit4]   2> 730898 T4209 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730898 T4221 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730898 T4241 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730898 T4207 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4251 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730898 T4237 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4233 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4243 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4245 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4213 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4229 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4231 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4249 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4191 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4239 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4227 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730898 T4235 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730901 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730900 T4203 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4197 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4215 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4223 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4217 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4225 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4211 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4205 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730900 T4199 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730903 T4253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17bf19d name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730899 T4201 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4247 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730899 T4219 oascc.ZkStateReader$3.process Updating live nodes... (28)
   [junit4]   2> 730903 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730904 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730905 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node28
   [junit4]   2> 730908 T4201 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730908 T4231 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730908 T4223 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4239 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4235 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4237 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4227 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4241 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730908 T4191 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4219 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4197 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730910 T4221 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4249 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4199 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4253 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4211 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730910 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730909 T4207 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4233 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4203 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4229 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730909 T4225 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730908 T4247 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730908 T4217 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730912 T4255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@caa414 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730910 T4243 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730910 T4245 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730910 T4213 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730910 T4205 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730910 T4251 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730910 T4209 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730910 T4215 oascc.ZkStateReader$3.process Updating live nodes... (29)
   [junit4]   2> 730912 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730914 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730915 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node29
   [junit4]   2> 730918 T4221 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730918 T4249 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730918 T4201 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730918 T4243 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730918 T4223 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730919 T4239 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730919 T4235 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730919 T4237 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730920 T4225 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730920 T4209 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730920 T4217 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4207 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4205 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4227 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730920 T4219 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730920 T4251 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730920 T4215 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730920 T4213 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730920 T4231 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730920 T4247 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730922 T4233 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730922 T4257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ed0605 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730921 T4245 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4191 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4255 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4211 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4203 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4229 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4197 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4253 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4241 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730921 T4199 oascc.ZkStateReader$3.process Updating live nodes... (30)
   [junit4]   2> 730923 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730925 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730926 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node30
   [junit4]   2> 730929 T4241 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730929 T4215 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730929 T4233 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4205 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4225 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730929 T4235 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4211 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4249 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4257 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4223 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730929 T4255 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730929 T4245 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4213 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4203 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4217 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4201 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4231 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4227 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4243 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4229 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4253 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4221 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4209 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4207 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4251 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4219 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4237 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730930 T4197 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730933 T4259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17a9421 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730932 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730931 T4239 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4199 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4191 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730931 T4247 oascc.ZkStateReader$3.process Updating live nodes... (31)
   [junit4]   2> 730934 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730935 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730937 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node31
   [junit4]   2> 730939 T4245 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4255 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4241 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4219 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4215 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4243 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4223 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4235 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4239 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4253 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4247 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4209 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4191 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4257 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4221 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4229 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4225 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4213 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4197 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4211 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4233 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730940 T4231 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730943 T4261 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1991915 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730942 T4259 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730941 T4199 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4205 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4237 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4227 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4203 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4249 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4201 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4217 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4207 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730941 T4251 oascc.ZkStateReader$3.process Updating live nodes... (32)
   [junit4]   2> 730944 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730946 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730948 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node32
   [junit4]   2> 730951 T4239 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730951 T4199 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4247 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4213 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4227 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4223 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4221 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4251 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4253 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730954 T4233 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4201 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4217 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4203 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730951 T4191 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730954 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730954 T4261 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730954 T4259 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730954 T4241 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730955 T4263 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13f37e5 name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730953 T4257 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4197 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730954 T4205 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4235 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4219 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4245 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4255 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4225 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4211 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4215 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4209 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730953 T4237 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4207 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4243 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4231 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4249 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730952 T4229 oascc.ZkStateReader$3.process Updating live nodes... (33)
   [junit4]   2> 730956 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 730959 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 730961 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node33
   [junit4]   2> 730964 T4203 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730964 T4191 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4239 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730964 T4223 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4199 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4229 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4217 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730966 T4251 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4243 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730966 T4221 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730966 T4209 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4207 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730966 T4219 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730967 T4211 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4249 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4227 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730967 T4255 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730967 T4263 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4231 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4247 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4213 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730965 T4201 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730968 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 730967 T4235 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730967 T4259 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730967 T4241 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730969 T4265 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16fdc8a name:ZooKeeperConnection Watcher:127.0.0.1:40590/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 730967 T4257 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730966 T4261 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730966 T4245 oascc.ZkStateReader$3.process Updating live nodes... (34)
   [junit4]   2> 730966 T4233 oascc.ZkStateReader$3.process Updating l

[...truncated too long message...]

c.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 770816 T4180 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 770820 T4711 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 770820 T4180 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 770821 T4712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 770822 T4703 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 770822 T4709 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 770823 T4711 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 770823 T4180 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 770824 T4715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@169d380 name:ZooKeeperConnection Watcher:127.0.0.1:44053/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 770824 T4180 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 770825 T4180 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 770826 T4180 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 770828 T4703 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 770828 T4715 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 770829 T4711 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 770829 T4180 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 770830 T4712 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 770830 T4712 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 770831 T4711 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 770832 T4180 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 770835 T4711 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 770837 T4711 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 770938 T4703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 770939 T4180 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 770938 T4715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 770942 T4715 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 770942 T4703 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 770943 T4180 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44053 44053
   [junit4]   2> 771021 T4697 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:44053 44053
   [junit4]   2> 771023 T4180 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
   [junit4]   2> 771024 T4712 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44053_solr
   [junit4]   2> 771024 T4716 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:302)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:291)
   [junit4]   2> 
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 771025 T4180 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 771025 T4180 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 774026 T4180 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J0/./solr.cloud.OverseerTest-1D2C2210D9BE7786-001
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=vi_VN, timezone=Asia/Taipei
   [junit4]   2> NOTE: Linux 3.8.0-38-generic i386/Oracle Corporation 1.7.0_60-ea (32-bit)/cpus=8,threads=1,free=176485264,total=309592064
   [junit4]   2> NOTE: All tests run in this JVM: [BadComponentTest, TestReplicationHandler, DistributedQueryElevationComponentTest, TestLuceneMatchVersion, TestReloadAndDeleteDocs, SuggesterFSTTest, TestRemoteStreaming, TestInitQParser, TestRestManager, CursorPagingTest, TestPHPSerializedResponseWriter, ShardRoutingTest, BJQParserTest, QueryParsingTest, LeaderElectionTest, TestLRUCache, TestNonNRTOpen, TestWriterPerf, TestLFUCache, TestNonDefinedSimilarityFactory, MBeansHandlerTest, PolyFieldTest, TestQueryUtils, SuggesterTest, TermsComponentTest, ScriptEngineTest, TestPseudoReturnFields, TestConfig, TestNRTOpen, OverseerRolesTest, TestSerializedLuceneMatchVersion, HdfsBasicDistributedZk2Test, HdfsRecoveryZkTest, WordBreakSolrSpellCheckerTest, SOLR749Test, ClusterStateUpdateTest, ShowFileRequestHandlerTest, TestConfigSets, TestExpandComponent, SpatialFilterTest, TestHighFrequencyDictionaryFactory, DocValuesTest, TestSolrXMLSerializer, SchemaVersionSpecificBehaviorTest, QueryElevationComponentTest, RecoveryZkTest, TestShortCircuitedRequests, TestStressUserVersions, TestCollapseQParserPlugin, CoreAdminHandlerTest, EchoParamsTest, TestDocSet, UpdateParamsTest, TestFieldCollectionResource, FastVectorHighlighterTest, URLClassifyProcessorTest, FieldMutatingUpdateProcessorTest, CircularListTest, UpdateRequestProcessorFactoryTest, TestValueSourceCache, TestSearchPerf, AutoCommitTest, BadCopyFieldTest, TestLMDirichletSimilarityFactory, TestBlendedInfixSuggestions, DOMUtilTest, FileBasedSpellCheckerTest, MoreLikeThisHandlerTest, ZkCLITest, UUIDFieldTest, TestSystemIdResolver, UnloadDistributedZkTest, TestSolrQueryParserDefaultOperatorResource, SimpleFacetsTest, TestSort, TestCoreContainer, TestStressVersions, SSLMigrationTest, DistributedQueryComponentCustomSortTest, TestManagedSchemaFieldResource, TestFreeTextSuggestions, DeleteReplicaTest, TestDistributedMissingSort, PrimitiveFieldTypeTest, ParsingFieldUpdateProcessorsTest, SynonymTokenizerTest, TestGroupingSearch, TestSimpleQParserPlugin, ExternalFileFieldSortTest, TestMergePolicyConfig, ZkControllerTest, SyncSliceTest, CSVRequestHandlerTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, OverseerTest]
   [junit4] Completed on J0 in 48.83s, 9 tests, 1 error <<< FAILURES!

[...truncated 749 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/lucene/common-build.xml:1293: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/lucene/common-build.xml:919: There were test failures: 387 suites, 1619 tests, 1 error, 37 ignored (16 assumptions)

Total time: 59 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_60-ea-b14 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message