lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.6.0_45) - Build # 8819 - Failure!
Date Wed, 01 Jan 2014 22:47:10 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/8819/
Java: 64bit/jdk1.6.0_45 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testOverseerFailure

Error Message:
KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1

Stack Trace:
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
	at __randomizedtesting.SeedInfo.seed([7F05D08FBF485B61:7B0D5F7CADEDB440]:0)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:428)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:425)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:157)
	at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
	at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:153)
	at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:584)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)




Build Log:
[...truncated 9935 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> 721541 T2756 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-OverseerTest-1388615820015
   [junit4]   2> 721542 T2756 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 721544 T2756 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
   [junit4]   2> 721545 T2756 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 721545 T2757 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 721645 T2756 oasc.ZkTestServer.run start zk server on port:53284
   [junit4]   2> 721646 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 721833 T2764 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6825aff6 name:ZooKeeperConnection Watcher:127.0.0.1:53284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 721833 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 721834 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 721837 T2767 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47019902 name:ZooKeeperConnection Watcher:127.0.0.1:53284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 721837 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 721840 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 721841 T2770 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f332eaf name:ZooKeeperConnection Watcher:127.0.0.1:53284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 721841 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 721841 T2756 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 721848 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 721852 T2756 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 721856 T2756 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 721859 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 721864 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 721866 T2773 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56d6a93c name:ZooKeeperConnection Watcher:127.0.0.1:53284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 721866 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 721869 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 721873 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 721878 T2764 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 721878 T2773 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 721880 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 721881 T2776 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d9fa894 name:ZooKeeperConnection Watcher:127.0.0.1:53284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 721881 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 721883 T2756 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 721887 T2756 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 721893 T2756 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 721896 T2756 oasc.Overseer.start Overseer (id=91004326380896260-127.0.0.1:53284_solr-n_0000000000) starting
   [junit4]   2> 721899 T2756 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 721904 T2756 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 721909 T2756 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 721913 T2778 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 721916 T2777 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 721919 T2777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 721919 T2777 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"recovering",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"12",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 721920 T2777 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5, shard6, shard7, shard8, shard9, shard10, shard11, shard12]
   [junit4]   2> 721920 T2777 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 721923 T2776 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 721924 T2764 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 721924 T2773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 722417 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 722429 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 722440 T2776 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722442 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 722444 T2777 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 722447 T2764 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 722447 T2773 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 722450 T2756 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53284 53284
   [junit4]   2> 722552 T2756 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 722555 T2756 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
   [junit4]   2> 722556 T2756 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 722556 T2779 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 722656 T2756 oasc.ZkTestServer.run start zk server on port:43357
   [junit4]   2> 722657 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 722661 T2786 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b72523a name:ZooKeeperConnection Watcher:127.0.0.1:43357 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 722661 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 722699 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 722702 T2789 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e035b12 name:ZooKeeperConnection Watcher:127.0.0.1:43357 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 722702 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 722702 T2756 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 722706 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 722708 T2792 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23ed4177 name:ZooKeeperConnection Watcher:127.0.0.1:43357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 722708 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 722708 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 722711 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 722713 T2795 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@663328d8 name:ZooKeeperConnection Watcher:127.0.0.1:43357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 722713 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 722715 T2756 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 722719 T2756 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 722725 T2756 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 722729 T2756 oasc.Overseer.start Overseer (id=91004326447349763-127.0.0.1:43357_solr-n_0000000000) starting
   [junit4]   2> 722732 T2756 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 722737 T2756 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 722743 T2756 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 722747 T2797 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 722748 T2756 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 722750 T2796 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 722752 T2756 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 722754 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 722758 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 722760 T2800 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@130826d7 name:ZooKeeperConnection Watcher:127.0.0.1:43357/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 722760 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 722762 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 722765 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
   [junit4]   2> 722770 T2792 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 722770 T2800 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 722773 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722777 T2796 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 722778 T2796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 722778 T2796 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 722779 T2796 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 722783 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 722784 T2800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 722784 T2792 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 723273 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 723286 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 723295 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 723298 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 723298 T2796 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 723306 T2796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 723306 T2796 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 723306 T2796 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 723309 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 723413 T2792 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 723414 T2800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 723798 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 723808 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 723815 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 723815 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 723815 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 723818 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 723819 T2796 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 723824 T2796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "core_node_name":"node3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 723824 T2796 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 723825 T2796 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 723829 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 723933 T2800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 723933 T2792 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 724318 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 724331 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 724338 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 724338 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 724338 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 724341 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 724342 T2796 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 724347 T2796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node4",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 724347 T2796 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 724347 T2796 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 724350 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 724453 T2800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 724453 T2792 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 724842 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 724854 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 724854 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 724854 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 724856 T2796 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 724857 T2796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core5",
   [junit4]   2> 	  "core_node_name":"node5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 724857 T2796 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 724857 T2796 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 724861 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 724965 T2800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 724966 T2792 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 725354 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 725365 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 725366 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 725366 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 725368 T2796 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 725369 T2796 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1",
   [junit4]   2> 	  "core":"core6",
   [junit4]   2> 	  "core_node_name":"node6",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
   [junit4]   2> 725369 T2796 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 725369 T2796 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 725372 T2795 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 725475 T2800 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 725476 T2792 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 725866 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 725883 T2800 oascc.ZkStateReader$3.process Updating live nodes... (0)
   [junit4]   2> 725888 T2756 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:43357 43357
   [junit4]   2> 726016 T2756 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 726022 T2756 oas.SolrTestCaseJ4.setUp ###Starting testReplay
   [junit4]   2> 726022 T2756 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 726023 T2801 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 726123 T2756 oasc.ZkTestServer.run start zk server on port:35774
   [junit4]   2> 726124 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726127 T2808 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@446bba9e name:ZooKeeperConnection Watcher:127.0.0.1:35774/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726127 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726128 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726130 T2811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b93f34c name:ZooKeeperConnection Watcher:127.0.0.1:35774 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726130 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726163 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726164 T2814 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4df1730e name:ZooKeeperConnection Watcher:127.0.0.1:35774 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726165 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726166 T2756 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 726172 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 726178 T2756 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 726185 T2756 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 726189 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 726195 T2756 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 726202 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726204 T2817 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72fc87a8 name:ZooKeeperConnection Watcher:127.0.0.1:35774/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726205 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726207 T2756 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 726211 T2756 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 726216 T2756 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 726219 T2756 oasc.Overseer.start Overseer (id=91004326674825219-127.0.0.1:35774_solr-n_0000000000) starting
   [junit4]   2> 726222 T2756 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 726228 T2756 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 726234 T2819 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 726239 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 726240 T2818 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 726240 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 726241 T2808 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 726247 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core2",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 726249 T2808 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 726252 T2818 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 726255 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 726256 T2818 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "shard":"s1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"core3",
   [junit4]   2> 	  "roles":"",
   [junit4]   2> 	  "state":"recovering"}
   [junit4]   2> 726259 T2817 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 726260 T2808 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 726341 T2756 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35774 35774
   [junit4]   2> 726485 T2756 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 726489 T2756 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
   [junit4]   2> 726490 T2756 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 726490 T2820 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 726590 T2756 oasc.ZkTestServer.run start zk server on port:60994
   [junit4]   2> 726591 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726594 T2827 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d0a1672 name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726594 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726595 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726596 T2830 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c39166d name:ZooKeeperConnection Watcher:127.0.0.1:60994 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726596 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726616 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726617 T2833 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@186530a4 name:ZooKeeperConnection Watcher:127.0.0.1:60994 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726617 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726618 T2756 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 726621 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 726625 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726625 T2834 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726627 T2838 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c9bab06 name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726627 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726627 T2840 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ff92596 name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726627 T2834 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726628 T2756 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 726628 T2834 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 726632 T2756 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 726632 T2834 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 726635 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 726640 T2834 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 726640 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 726643 T2834 oasc.Overseer.start Overseer (id=91004326705168388-127.0.0.1:60994_solr-n_0000000000) starting
   [junit4]   2> 726644 T2838 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 726646 T2756 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 726647 T2834 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 726654 T2834 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 726659 T2834 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 726662 T2834 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 726662 T2842 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 726664 T2834 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 726665 T2845 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23f901b1 name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 726665 T2834 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 726669 T2834 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 726673 T2834 oasc.Overseer.start Overseer (id=91004326705168389-127.0.0.1:60994_solr-n_0000000001) starting
   [junit4]   2> 726682 T2847 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 726684 T2846 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 726688 T2846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 726689 T2846 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 726689 T2846 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
   [junit4]   2> 726689 T2846 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 726693 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 726694 T2838 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 727154 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 727169 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 727177 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727180 T2846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 727181 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727181 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 727188 T2846 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 727188 T2846 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 727193 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 727193 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727194 T2850 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4779f245 name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 727194 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 727196 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 727199 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 727204 T2838 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 727205 T2850 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 727208 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727208 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 727210 T2846 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 727211 T2846 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 727215 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727219 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 727318 T2850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 727319 T2838 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 727384 T2800 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 727385 T2800 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 727385 T2800 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:93)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 727421 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 727426 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727426 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727426 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727428 T2846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 727429 T2846 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 727429 T2846 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
   [junit4]   2> 727429 T2846 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 727432 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727535 T2850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 727535 T2838 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 727926 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 727936 T2850 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 727936 T2838 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 727939 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 727939 T2850 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 727948 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727948 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727948 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 727950 T2846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 727955 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728058 T2850 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 728141 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728143 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 728145 T2853 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e1db6bd name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 728145 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 728147 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 728150 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 728155 T2850 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 728155 T2853 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 728158 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728159 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728159 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 728159 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728163 T2846 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728163 T2841 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer work queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue-work
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.element(DistributedQueue.java:131)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:325)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:105)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> 
   [junit4]   2> 728164 T2842 oasc.OverseerCollectionProcessor.amILeader  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:270)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:270)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:204)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:163)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> 
   [junit4]   2> 728165 T2842 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=91004326705168388-127.0.0.1:60994_solr-n_0000000000) am no longer a leader.
   [junit4]   2> 728166 T2846 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 728166 T2846 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728170 T2845 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728174 T2853 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 728174 T2850 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 728187 T2834 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 728187 T2853 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 728189 T2834 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 728189 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 728191 T2856 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f4055a1 name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 728192 T2834 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 728197 T2834 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 728201 T2834 oasc.Overseer.start Overseer (id=91004326705168392-127.0.0.1:60994_solr-n_0000000002) starting
   [junit4]   2> 728204 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 728205 T2859 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@188ab40f name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 728205 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 728207 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 728209 T2861 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 728213 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 728213 T2860 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728214 T2860 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 728215 T2860 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 728216 T2860 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728218 T2853 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 728219 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728220 T2859 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 728220 T2859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728223 T2860 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 728225 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 728227 T2860 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728228 T2860 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 728228 T2860 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728232 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728233 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728233 T2859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728236 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728241 T2860 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728244 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728245 T2860 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 728246 T2860 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728248 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728272 T2846 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:411)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:195)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> 
   [junit4]   2> 728352 T2853 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728352 T2859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728439 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728445 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728445 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728445 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 728445 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728449 T2860 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728450 T2860 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 728450 T2860 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728456 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728460 T2859 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 728460 T2853 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 728463 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728463 T2859 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 728469 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728473 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728576 T2859 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 728665 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728667 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 728668 T2864 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65e86f47 name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 728669 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 728670 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 728674 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 728678 T2859 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 728678 T2864 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 728681 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 728681 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728682 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728682 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728685 T2860 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728686 T2860 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 728686 T2860 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728692 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728696 T2864 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 728696 T2859 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 728699 T2864 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 728700 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728701 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 728702 T2860 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 728702 T2860 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728706 T2856 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728709 T2834 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 728710 T2834 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 728712 T2867 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f1cddfb name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 728712 T2834 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 728714 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 728716 T2870 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@414e8cfc name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 728716 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 728717 T2834 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 728718 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 728721 T2834 oasc.Overseer.start Overseer (id=91004326705168395-127.0.0.1:60994_solr-n_0000000003) starting
   [junit4]   2> 728723 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 728728 T2864 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 728728 T2870 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 728729 T2872 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 728732 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 728734 T2871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728735 T2871 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 728736 T2871 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 728736 T2871 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728738 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728738 T2864 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728745 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728745 T2871 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 728746 T2871 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728747 T2864 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728747 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728750 T2871 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 728753 T2871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728756 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728757 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728757 T2864 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728764 T2871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728765 T2871 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 728765 T2871 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728768 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728808 T2860 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:89)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:411)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:195)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
   [junit4]   2> 
   [junit4]   2> 728871 T2864 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728872 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 728947 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728951 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728952 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728952 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728952 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 728955 T2871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728956 T2871 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node2",
   [junit4]   2> 	  "core":"core4",
   [junit4]   2> 	  "core_node_name":"node2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node2/solr/"}
   [junit4]   2> 728956 T2871 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 728962 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728967 T2870 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 728967 T2864 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 728970 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 728970 T2870 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 728978 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 728983 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729085 T2870 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 729172 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 729174 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 729175 T2875 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68787071 name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 729176 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 729177 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 729182 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
   [junit4]   2> 729187 T2870 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 729187 T2875 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 729190 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729190 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729190 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 729190 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729194 T2871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 729195 T2871 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state1",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 729195 T2871 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 729199 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729203 T2875 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 729203 T2870 oascc.ZkStateReader$3.process Updating live nodes... (1)
   [junit4]   2> 729206 T2875 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 729207 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729207 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 729211 T2871 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"state2",
   [junit4]   2> 	  "node_name":"node1",
   [junit4]   2> 	  "core":"core1",
   [junit4]   2> 	  "core_node_name":"node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://node1/solr/"}
   [junit4]   2> 729211 T2871 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
   [junit4]   2> 729215 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729220 T2756 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 729222 T2878 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d3ec691 name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 729222 T2756 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 729223 T2867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 729225 T2756 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 729228 T2756 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
   [junit4]   2> 729229 T2834 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
   [junit4]   2> 729231 T2834 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 729232 T2881 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6eea90ee name:ZooKeeperConnection Watcher:127.0.0.1:60994/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 729232 T2834 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 729233 T2875 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 729233 T2878 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 729237 T2756 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 729238 T2834 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 729242 T2834 oasc.Overseer.start Overseer (id=91004326705168399-127.0.0.1:60994_solr-n_0000000004) starting
   [junit4]   2> 729249 T2756 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 729250 T2883 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 729254 T2882 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 729254 T2882 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
   [junit4]   2> 729255 T2882 oasc.O

[...truncated too long message...]

olr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 751801 T3291 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 751819 T3300 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 751820 T3300 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 751826 T3306 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 751827 T3306 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 751829 T3309 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:165)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:109)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:138)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 751830 T3309 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 753432 T2756 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=hr, timezone=Asia/Urumqi
   [junit4]   2> NOTE: Linux 3.8.0-34-generic amd64/Sun Microsystems Inc. 1.6.0_45 (64-bit)/cpus=8,threads=1,free=84521488,total=162230272
   [junit4]   2> NOTE: All tests run in this JVM: [TestNonNRTOpen, ClusterStateUpdateTest, TestPerFieldSimilarity, AnalysisAfterCoreReloadTest, SolrTestCaseJ4Test, TestSort, TestCollationFieldDocValues, CacheHeaderTest, FieldMutatingUpdateProcessorTest, SoftAutoCommitTest, QueryEqualityTest, TestCloudManagedSchema, EchoParamsTest, TestPHPSerializedResponseWriter, TestBadConfig, DistributedDebugComponentTest, TestFaceting, ReturnFieldsTest, TestQueryUtils, SpellPossibilityIteratorTest, TestJoin, HdfsLockFactoryTest, BinaryUpdateRequestHandlerTest, TestSchemaNameResource, TestNRTOpen, TestSerializedLuceneMatchVersion, TestSolrQueryParser, StatelessScriptUpdateProcessorFactoryTest, SchemaVersionSpecificBehaviorTest, TestCopyFieldCollectionResource, URLClassifyProcessorTest, DeleteReplicaTest, TestDynamicFieldResource, CircularListTest, DocValuesMissingTest, TestSolrIndexConfig, TestReloadAndDeleteDocs, BadCopyFieldTest, DeleteShardTest, UnloadDistributedZkTest, ResourceLoaderTest, LoggingHandlerTest, BJQParserTest, TestRandomMergePolicy, BadComponentTest, TestRemoteStreaming, TestDocSet, CoreAdminCreateDiscoverTest, UpdateParamsTest, SolrIndexSplitterTest, SolrXmlInZkTest, SampleTest, TestStressVersions, TestFunctionQuery, TestLuceneMatchVersion, TestDocBasedVersionConstraints, TestPostingsSolrHighlighter, TestRecoveryHdfs, TestManagedSchema, ShardRoutingTest, WordBreakSolrSpellCheckerTest, TestCharFilters, HdfsDirectoryTest, TestMergePolicyConfig, HdfsUnloadDistributedZkTest, TestSolrXMLSerializer, OverseerCollectionProcessorTest, FileBasedSpellCheckerTest, ZkCLITest, DistributedSpellCheckComponentTest, HdfsBasicDistributedZk2Test, CoreAdminHandlerTest, SuggesterTest, PeerSyncTest, TestCoreContainer, AutoCommitTest, TestFileDictionaryLookup, RegexBoostProcessorTest, TestFieldTypeCollectionResource, TestLMDirichletSimilarityFactory, DirectUpdateHandlerOptimizeTest, QueryParsingTest, TestGroupingSearch, TestReversedWildcardFilterFactory, BasicDistributedZkTest, SyncSliceTest, OverseerTest]
   [junit4] Completed on J0 in 31.93s, 8 tests, 1 error <<< FAILURES!

[...truncated 535 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:459: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:439: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:491: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1307: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:940: There were test failures: 348 suites, 1522 tests, 1 error, 36 ignored (7 assumptions)

Total time: 45 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0_45 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message