lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@sd-datasolutions.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (64bit/ibm-j9-jdk6) - Build # 2135 - Failure!
Date Thu, 01 Nov 2012 16:09:57 GMT
Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Linux/2135/
Java: 64bit/ibm-j9-jdk6 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testShardLeaderChange

Error Message:
Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>

Stack Trace:
org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>
	at __randomizedtesting.SeedInfo.seed([DA6A5DCD24EF6E3B:439DA3A3E779BCA]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:493)
	at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:652)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
	at java.lang.reflect.Method.invoke(Method.java:611)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:738)




Build Log:
[...truncated 7822 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4:junit4]   2> 0 T555 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351785420697
[junit4:junit4]   2> 1 T555 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 5 T555 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4:junit4]   2> 5 T555 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 8 T556 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 8 T556 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 9 T556 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 106 T555 oasc.ZkTestServer.run start zk server on port:42858
[junit4:junit4]   2> 107 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42858/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@45784578
[junit4:junit4]   2> 108 T561 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42858
[junit4:junit4]   2> 108 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 109 T561 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42858, initiating session
[junit4:junit4]   2> 110 T557 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42576
[junit4:junit4]   2> 110 T557 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42576
[junit4:junit4]   2> 111 T559 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 198 T559 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb097a20000 with negotiated timeout 10000 for client /127.0.0.1:42576
[junit4:junit4]   2> 199 T561 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42858, sessionid = 0x13abcb097a20000, negotiated timeout = 10000
[junit4:junit4]   2> 199 T562 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45784578 name:ZooKeeperConnection Watcher:127.0.0.1:42858/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 200 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 201 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42858 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@78ae78ae
[junit4:junit4]   2> 202 T563 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42858
[junit4:junit4]   2> 202 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 203 T563 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42858, initiating session
[junit4:junit4]   2> 203 T557 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42577
[junit4:junit4]   2> 204 T557 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42577
[junit4:junit4]   2> 206 T563 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42858, sessionid = 0x13abcb097a20001, negotiated timeout = 10000
[junit4:junit4]   2> 206 T559 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb097a20001 with negotiated timeout 10000 for client /127.0.0.1:42577
[junit4:junit4]   2> 206 T564 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78ae78ae name:ZooKeeperConnection Watcher:127.0.0.1:42858 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 218 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 218 T560 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb097a20001
[junit4:junit4]   2> 219 T564 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 219 T555 oaz.ZooKeeper.close Session: 0x13abcb097a20001 closed
[junit4:junit4]   2> 219 T557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:42577 which had sessionid 0x13abcb097a20001
[junit4:junit4]   2> 221 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42858 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4f994f99
[junit4:junit4]   2> 222 T565 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42858
[junit4:junit4]   2> 222 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 223 T565 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42858, initiating session
[junit4:junit4]   2> 223 T557 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42578
[junit4:junit4]   2> 224 T557 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42578
[junit4:junit4]   2> 225 T565 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42858, sessionid = 0x13abcb097a20002, negotiated timeout = 10000
[junit4:junit4]   2> 225 T559 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb097a20002 with negotiated timeout 10000 for client /127.0.0.1:42578
[junit4:junit4]   2> 226 T566 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f994f99 name:ZooKeeperConnection Watcher:127.0.0.1:42858 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 227 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 228 T555 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 229 T560 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb097a20002
[junit4:junit4]   2> 230 T566 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 230 T555 oaz.ZooKeeper.close Session: 0x13abcb097a20002 closed
[junit4:junit4]   2> 230 T557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:42578 which had sessionid 0x13abcb097a20002
[junit4:junit4]   2> 231 T555 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 233 T555 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 234 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 235 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42858/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4c004c0
[junit4:junit4]   2> 236 T567 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42858
[junit4:junit4]   2> 236 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 237 T567 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42858, initiating session
[junit4:junit4]   2> 237 T557 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42579
[junit4:junit4]   2> 238 T557 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42579
[junit4:junit4]   2> 239 T567 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42858, sessionid = 0x13abcb097a20003, negotiated timeout = 10000
[junit4:junit4]   2> 239 T559 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb097a20003 with negotiated timeout 10000 for client /127.0.0.1:42579
[junit4:junit4]   2> 240 T568 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c004c0 name:ZooKeeperConnection Watcher:127.0.0.1:42858/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 241 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 242 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 244 T560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb097a20003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 245 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 248 T555 oasc.Overseer.start Overseer (id=88590609331388419-127.0.0.1:42858_solr-n_0000000000) starting
[junit4:junit4]   2> 250 T560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb097a20003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 251 T560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb097a20003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 252 T560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb097a20003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 253 T570 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 253 T560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb097a20000 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 254 T569 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 254 T560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb097a20000 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 256 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 307 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 358 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 410 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 461 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 512 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 565 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 617 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 668 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 719 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 757 T569 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 758 T569 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 759 T560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb097a20003 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 762 T562 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 770 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1264 T569 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1265 T569 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 1268 T562 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 1273 T560 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb097a20000
[junit4:junit4]   2> 1274 T562 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1274 T555 oaz.ZooKeeper.close Session: 0x13abcb097a20000 closed
[junit4:junit4]   2> 1274 T557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:42576 which had sessionid 0x13abcb097a20000
[junit4:junit4]   2> 1275 T560 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb097a20003
[junit4:junit4]   2> 1276 T568 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1276 T555 oaz.ZooKeeper.close Session: 0x13abcb097a20003 closed
[junit4:junit4]   2> 1277 T557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:42579 which had sessionid 0x13abcb097a20003
[junit4:junit4]   2> 1277 T560 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 1277 T559 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 1279 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 1280 T557 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 1281 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 1281 T555 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4:junit4]   2> 1288 T555 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
[junit4:junit4]   2> 1288 T555 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 1289 T571 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 1290 T571 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 1291 T571 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351785420697/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 1293 T571 oazsp.FileTxnSnapLog.save Snapshotting: 24
[junit4:junit4]   2> 1304 T558 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 1389 T555 oasc.ZkTestServer.run start zk server on port:43820
[junit4:junit4]   2> 1390 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43820/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@39943994
[junit4:junit4]   2> 1391 T576 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:43820
[junit4:junit4]   2> 1391 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1392 T576 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43820, initiating session
[junit4:junit4]   2> 1392 T572 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60422
[junit4:junit4]   2> 1393 T572 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60422
[junit4:junit4]   2> 1393 T574 oazsp.FileTxnLog.append Creating new log file: log.25
[junit4:junit4]   2> 1396 T574 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09ca60000 with negotiated timeout 10000 for client /127.0.0.1:60422
[junit4:junit4]   2> 1396 T576 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:43820, sessionid = 0x13abcb09ca60000, negotiated timeout = 10000
[junit4:junit4]   2> 1397 T577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39943994 name:ZooKeeperConnection Watcher:127.0.0.1:43820/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1397 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1398 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43820 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@69396939
[junit4:junit4]   2> 1399 T578 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:43820
[junit4:junit4]   2> 1399 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1400 T578 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43820, initiating session
[junit4:junit4]   2> 1400 T572 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60423
[junit4:junit4]   2> 1401 T572 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60423
[junit4:junit4]   2> 1402 T574 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09ca60001 with negotiated timeout 10000 for client /127.0.0.1:60423
[junit4:junit4]   2> 1402 T578 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:43820, sessionid = 0x13abcb09ca60001, negotiated timeout = 10000
[junit4:junit4]   2> 1403 T579 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69396939 name:ZooKeeperConnection Watcher:127.0.0.1:43820 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1404 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1409 T575 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09ca60001
[junit4:junit4]   2> 1410 T579 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1410 T555 oaz.ZooKeeper.close Session: 0x13abcb09ca60001 closed
[junit4:junit4]   2> 1410 T572 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60423 which had sessionid 0x13abcb09ca60001
[junit4:junit4]   2> 1411 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43820 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@c230c23
[junit4:junit4]   2> 1412 T580 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:43820
[junit4:junit4]   2> 1412 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1413 T580 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43820, initiating session
[junit4:junit4]   2> 1413 T572 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60424
[junit4:junit4]   2> 1414 T572 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60424
[junit4:junit4]   2> 1415 T580 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:43820, sessionid = 0x13abcb09ca60002, negotiated timeout = 10000
[junit4:junit4]   2> 1415 T574 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09ca60002 with negotiated timeout 10000 for client /127.0.0.1:60424
[junit4:junit4]   2> 1415 T581 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c230c23 name:ZooKeeperConnection Watcher:127.0.0.1:43820 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1416 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1417 T555 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 1418 T575 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09ca60002
[junit4:junit4]   2> 1419 T581 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1419 T555 oaz.ZooKeeper.close Session: 0x13abcb09ca60002 closed
[junit4:junit4]   2> 1419 T572 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60424 which had sessionid 0x13abcb09ca60002
[junit4:junit4]   2> 1420 T555 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1421 T555 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1422 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1423 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43820/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@69c969c9
[junit4:junit4]   2> 1424 T582 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:43820
[junit4:junit4]   2> 1424 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1425 T582 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43820, initiating session
[junit4:junit4]   2> 1425 T572 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60425
[junit4:junit4]   2> 1466 T572 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60425
[junit4:junit4]   2> 1467 T582 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:43820, sessionid = 0x13abcb09ca60003, negotiated timeout = 10000
[junit4:junit4]   2> 1467 T574 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09ca60003 with negotiated timeout 10000 for client /127.0.0.1:60425
[junit4:junit4]   2> 1468 T583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69c969c9 name:ZooKeeperConnection Watcher:127.0.0.1:43820/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1468 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1469 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1470 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 1471 T577 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 1471 T583 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 1472 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:43820/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@59355935
[junit4:junit4]   2> 1473 T584 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:43820
[junit4:junit4]   2> 1473 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1474 T584 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:43820, initiating session
[junit4:junit4]   2> 1474 T572 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60426
[junit4:junit4]   2> 1475 T572 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60426
[junit4:junit4]   2> 1476 T584 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:43820, sessionid = 0x13abcb09ca60004, negotiated timeout = 10000
[junit4:junit4]   2> 1476 T574 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09ca60004 with negotiated timeout 10000 for client /127.0.0.1:60426
[junit4:junit4]   2> 1476 T585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59355935 name:ZooKeeperConnection Watcher:127.0.0.1:43820/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1477 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1478 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1480 T575 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09ca60004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1480 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1482 T555 oasc.Overseer.start Overseer (id=88590609415536644-127.0.0.1:43820_solr-n_0000000000) starting
[junit4:junit4]   2> 1483 T575 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09ca60004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1483 T575 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09ca60004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1484 T575 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09ca60004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1485 T587 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1485 T575 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09ca60003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1486 T575 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09ca60003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 1487 T586 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1488 T586 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1488 T586 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "numShards":"12",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 1489 T586 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 12
[junit4:junit4]   2> 1489 T575 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09ca60004 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 1492 T577 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 1493 T583 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 1769 T569 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:738)
[junit4:junit4]   2> 
[junit4:junit4]   2> 1770 T569 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88590609331388419-127.0.0.1:42858_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 1988 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 1993 T575 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09ca60003 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 1994 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 1997 T575 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09ca60003 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1998 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2000 T575 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09ca60004
[junit4:junit4]   2> 2001 T585 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 2001 T572 oazs.NIOServerCnxn.doIO WARNING EndOfStreamException: Unable to read additional data from client sessionid 0x13abcb09ca60004, likely client has closed socket
[junit4:junit4]   2> 2002 T572 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60426 which had sessionid 0x13abcb09ca60004
[junit4:junit4]   2> 2000 T555 oaz.ZooKeeper.close Session: 0x13abcb09ca60004 closed
[junit4:junit4]   2> 2004 T577 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2004 T583 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2004 T575 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09ca60003
[junit4:junit4]   2> 2007 T572 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60425 which had sessionid 0x13abcb09ca60003
[junit4:junit4]   2> 2007 T555 oaz.ZooKeeper.close Session: 0x13abcb09ca60003 closed
[junit4:junit4]   2> 2010 T575 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09ca60000
[junit4:junit4]   2> 2011 T555 oaz.ZooKeeper.close Session: 0x13abcb09ca60000 closed
[junit4:junit4]   2> 2012 T572 oazs.NIOServerCnxn.doIO WARNING EndOfStreamException: Unable to read additional data from client sessionid 0x13abcb09ca60000, likely client has closed socket
[junit4:junit4]   2> 2012 T572 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60422 which had sessionid 0x13abcb09ca60000
[junit4:junit4]   2> 2013 T575 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 2014 T574 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 2014 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 2017 T572 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 2018 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 2019 T555 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
[junit4:junit4]   2> 2029 T555 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
[junit4:junit4]   2> 2030 T555 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 2031 T588 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 2032 T588 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 2034 T588 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351785420697/zookeeper/server1/data/version-2/snapshot.24
[junit4:junit4]   2> 2038 T588 oazsp.FileTxnSnapLog.save Snapshotting: 5b
[junit4:junit4]   2> 2131 T555 oasc.ZkTestServer.run start zk server on port:50036
[junit4:junit4]   2> 2132 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50036/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@2e5e2e5e
[junit4:junit4]   2> 2133 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2134 T593 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50036
[junit4:junit4]   2> 2136 T589 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58025
[junit4:junit4]   2> 2136 T593 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50036, initiating session
[junit4:junit4]   2> 2137 T589 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58025
[junit4:junit4]   2> 2138 T591 oazsp.FileTxnLog.append Creating new log file: log.5c
[junit4:junit4]   2> 2148 T591 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09f8f0000 with negotiated timeout 10000 for client /127.0.0.1:58025
[junit4:junit4]   2> 2157 T593 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50036, sessionid = 0x13abcb09f8f0000, negotiated timeout = 10000
[junit4:junit4]   2> 2157 T594 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e5e2e5e name:ZooKeeperConnection Watcher:127.0.0.1:50036/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2158 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2159 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50036 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6ba66ba6
[junit4:junit4]   2> 2160 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2161 T595 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50036
[junit4:junit4]   2> 2162 T595 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50036, initiating session
[junit4:junit4]   2> 2162 T589 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58026
[junit4:junit4]   2> 2163 T589 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58026
[junit4:junit4]   2> 2165 T591 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09f8f0001 with negotiated timeout 10000 for client /127.0.0.1:58026
[junit4:junit4]   2> 2166 T595 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50036, sessionid = 0x13abcb09f8f0001, negotiated timeout = 10000
[junit4:junit4]   2> 2167 T596 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ba66ba6 name:ZooKeeperConnection Watcher:127.0.0.1:50036 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2168 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2190 T592 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09f8f0001
[junit4:junit4]   2> 2191 T596 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 2191 T555 oaz.ZooKeeper.close Session: 0x13abcb09f8f0001 closed
[junit4:junit4]   2> 2191 T589 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58026 which had sessionid 0x13abcb09f8f0001
[junit4:junit4]   2> 2192 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50036 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@29642964
[junit4:junit4]   2> 2194 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2207 T597 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50036
[junit4:junit4]   2> 2209 T597 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50036, initiating session
[junit4:junit4]   2> 2209 T589 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58027
[junit4:junit4]   2> 2210 T589 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58027
[junit4:junit4]   2> 2211 T591 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09f8f0002 with negotiated timeout 10000 for client /127.0.0.1:58027
[junit4:junit4]   2> 2221 T597 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50036, sessionid = 0x13abcb09f8f0002, negotiated timeout = 10000
[junit4:junit4]   2> 2221 T598 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29642964 name:ZooKeeperConnection Watcher:127.0.0.1:50036 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2222 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2223 T555 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 2226 T592 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09f8f0002
[junit4:junit4]   2> 2227 T598 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 2227 T555 oaz.ZooKeeper.close Session: 0x13abcb09f8f0002 closed
[junit4:junit4]   2> 2227 T589 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58027 which had sessionid 0x13abcb09f8f0002
[junit4:junit4]   2> 2228 T555 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 2231 T555 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 2233 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2234 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50036/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@26152615
[junit4:junit4]   2> 2235 T599 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50036
[junit4:junit4]   2> 2235 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2237 T589 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58028
[junit4:junit4]   2> 2237 T599 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50036, initiating session
[junit4:junit4]   2> 2242 T589 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58028
[junit4:junit4]   2> 2243 T591 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09f8f0003 with negotiated timeout 10000 for client /127.0.0.1:58028
[junit4:junit4]   2> 2243 T599 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50036, sessionid = 0x13abcb09f8f0003, negotiated timeout = 10000
[junit4:junit4]   2> 2257 T600 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26152615 name:ZooKeeperConnection Watcher:127.0.0.1:50036/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2257 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2258 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2261 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 2268 T594 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 2268 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50036/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@12481248
[junit4:junit4]   2> 2269 T600 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 2270 T601 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50036
[junit4:junit4]   2> 2270 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2271 T601 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50036, initiating session
[junit4:junit4]   2> 2272 T589 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58029
[junit4:junit4]   2> 2273 T589 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58029
[junit4:junit4]   2> 2291 T591 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09f8f0004 with negotiated timeout 10000 for client /127.0.0.1:58029
[junit4:junit4]   2> 2291 T601 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50036, sessionid = 0x13abcb09f8f0004, negotiated timeout = 10000
[junit4:junit4]   2> 2292 T602 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12481248 name:ZooKeeperConnection Watcher:127.0.0.1:50036/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2293 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2294 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 2297 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 2298 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 2300 T555 oasc.Overseer.start Overseer (id=88590609464360964-127.0.0.1:50036_solr-n_0000000000) starting
[junit4:junit4]   2> 2301 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2302 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2304 T573 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 2305 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2318 T604 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 2319 T603 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 2495 T586 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:738)
[junit4:junit4]   2> 
[junit4:junit4]   2> 2496 T586 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88590609415536644-127.0.0.1:43820_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 3005 T583 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 3006 T583 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 3006 T583 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 3010 T577 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 3011 T577 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 3306 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3308 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 3322 T603 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3323 T603 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 3324 T603 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 3324 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0004 type:create cxid:0x23 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 3329 T594 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3329 T600 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3810 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 3817 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 3817 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 3819 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3820 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3822 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3824 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 3828 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0x2e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 3831 T603 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3835 T603 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 3838 T600 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3839 T594 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3889 T592 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09f8f0004
[junit4:junit4]   2> 3891 T602 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 3891 T555 oaz.ZooKeeper.close Session: 0x13abcb09f8f0004 closed
[junit4:junit4]   2> 3891 T589 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58029 which had sessionid 0x13abcb09f8f0004
[junit4:junit4]   2> 4342 T603 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:738)
[junit4:junit4]   2> 
[junit4:junit4]   2> 4343 T603 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88590609464360964-127.0.0.1:50036_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 4893 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0x34 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4895 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 4896 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0x3b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 4899 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50036/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@34d834d8
[junit4:junit4]   2> 4900 T605 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50036
[junit4:junit4]   2> 4900 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4901 T605 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50036, initiating session
[junit4:junit4]   2> 4901 T589 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58033
[junit4:junit4]   2> 4902 T589 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58033
[junit4:junit4]   2> 4903 T591 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb09f8f0005 with negotiated timeout 10000 for client /127.0.0.1:58033
[junit4:junit4]   2> 4903 T605 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50036, sessionid = 0x13abcb09f8f0005, negotiated timeout = 10000
[junit4:junit4]   2> 4903 T606 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34d834d8 name:ZooKeeperConnection Watcher:127.0.0.1:50036/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4904 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4905 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 4906 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 4908 T555 oasc.Overseer.start Overseer (id=88590609464360965-127.0.0.1:50036_solr-n_0000000002) starting
[junit4:junit4]   2> 4908 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0005 type:create cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4909 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4910 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0005 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4911 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0005 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4912 T608 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 4912 T607 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 4914 T607 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4915 T607 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 4918 T594 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 4918 T600 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 4969 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0x41 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4971 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 4973 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0x48 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 5420 T607 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5424 T600 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 5426 T592 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb09f8f0003 type:create cxid:0x54 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5428 T600 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 5429 T600 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 5433 T607 oasc.Overseer$ClusterStateUpdater.setShardLeader SEVERE Could not mark shard leader for non existing collection:collection1
[junit4:junit4]   2> 5435 T600 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 5435 T594 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 5936 T594 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 5936 T600 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 5936 T592 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09f8f0003
[junit4:junit4]   2> 5938 T555 oaz.ZooKeeper.close Session: 0x13abcb09f8f0003 closed
[junit4:junit4]   2> 5938 T589 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58028 which had sessionid 0x13abcb09f8f0003
[junit4:junit4]   2> 5939 T592 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09f8f0005
[junit4:junit4]   2> 5940 T594 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 5940 T589 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58033 which had sessionid 0x13abcb09f8f0005
[junit4:junit4]   2> 5940 T606 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 5940 T555 oaz.ZooKeeper.close Session: 0x13abcb09f8f0005 closed
[junit4:junit4]   2> 5942 T592 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb09f8f0000
[junit4:junit4]   2> 5942 T594 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 5942 T555 oaz.ZooKeeper.close Session: 0x13abcb09f8f0000 closed
[junit4:junit4]   2> 5943 T589 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58025 which had sessionid 0x13abcb09f8f0000
[junit4:junit4]   2> 5943 T592 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 5943 T591 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 5944 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 5946 T589 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 5947 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 5947 T555 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4:junit4]   2> 5961 T555 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
[junit4:junit4]   2> 5961 T555 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 5962 T609 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 5963 T609 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 5964 T609 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351785420697/zookeeper/server1/data/version-2/snapshot.5b
[junit4:junit4]   2> 5968 T609 oazsp.FileTxnSnapLog.save Snapshotting: ce
[junit4:junit4]   2> 6062 T555 oasc.ZkTestServer.run start zk server on port:42606
[junit4:junit4]   2> 6063 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42606/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@47a347a3
[junit4:junit4]   2> 6064 T614 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42606
[junit4:junit4]   2> 6064 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6065 T614 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42606, initiating session
[junit4:junit4]   2> 6065 T610 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58463
[junit4:junit4]   2> 6065 T610 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58463
[junit4:junit4]   2> 6066 T612 oazsp.FileTxnLog.append Creating new log file: log.cf
[junit4:junit4]   2> 6068 T612 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0aee90000 with negotiated timeout 10000 for client /127.0.0.1:58463
[junit4:junit4]   2> 6068 T614 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42606, sessionid = 0x13abcb0aee90000, negotiated timeout = 10000
[junit4:junit4]   2> 6069 T615 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47a347a3 name:ZooKeeperConnection Watcher:127.0.0.1:42606/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6069 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6070 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42606 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@37da37da
[junit4:junit4]   2> 6071 T616 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42606
[junit4:junit4]   2> 6071 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6072 T616 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42606, initiating session
[junit4:junit4]   2> 6072 T610 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58464
[junit4:junit4]   2> 6073 T610 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58464
[junit4:junit4]   2> 6074 T612 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0aee90001 with negotiated timeout 10000 for client /127.0.0.1:58464
[junit4:junit4]   2> 6074 T616 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42606, sessionid = 0x13abcb0aee90001, negotiated timeout = 10000
[junit4:junit4]   2> 6074 T617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37da37da name:ZooKeeperConnection Watcher:127.0.0.1:42606 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6075 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6080 T613 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0aee90001
[junit4:junit4]   2> 6080 T555 oaz.ZooKeeper.close Session: 0x13abcb0aee90001 closed
[junit4:junit4]   2> 6081 T617 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 6081 T610 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58464 which had sessionid 0x13abcb0aee90001
[junit4:junit4]   2> 6081 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42606 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@306d306d
[junit4:junit4]   2> 6083 T618 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42606
[junit4:junit4]   2> 6083 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6084 T618 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42606, initiating session
[junit4:junit4]   2> 6084 T610 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58465
[junit4:junit4]   2> 6085 T610 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58465
[junit4:junit4]   2> 6086 T612 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0aee90002 with negotiated timeout 10000 for client /127.0.0.1:58465
[junit4:junit4]   2> 6086 T618 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42606, sessionid = 0x13abcb0aee90002, negotiated timeout = 10000
[junit4:junit4]   2> 6087 T619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@306d306d name:ZooKeeperConnection Watcher:127.0.0.1:42606 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6087 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6087 T555 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 6089 T613 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0aee90002
[junit4:junit4]   2> 6089 T555 oaz.ZooKeeper.close Session: 0x13abcb0aee90002 closed
[junit4:junit4]   2> 6090 T610 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58465 which had sessionid 0x13abcb0aee90002
[junit4:junit4]   2> 6090 T555 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 6090 T619 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 6092 T555 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 6093 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 6094 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42606/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@36773677
[junit4:junit4]   2> 6094 T620 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42606
[junit4:junit4]   2> 6094 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6095 T620 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42606, initiating session
[junit4:junit4]   2> 6095 T610 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58466
[junit4:junit4]   2> 6096 T610 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58466
[junit4:junit4]   2> 6097 T612 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0aee90003 with negotiated timeout 10000 for client /127.0.0.1:58466
[junit4:junit4]   2> 6097 T620 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42606, sessionid = 0x13abcb0aee90003, negotiated timeout = 10000
[junit4:junit4]   2> 6098 T621 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36773677 name:ZooKeeperConnection Watcher:127.0.0.1:42606/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6098 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6099 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 6100 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 6101 T621 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 6101 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42606/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@22932293
[junit4:junit4]   2> 6102 T622 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42606
[junit4:junit4]   2> 6102 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6104 T622 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42606, initiating session
[junit4:junit4]   2> 6104 T610 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58467
[junit4:junit4]   2> 6105 T610 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58467
[junit4:junit4]   2> 6106 T612 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0aee90004 with negotiated timeout 10000 for client /127.0.0.1:58467
[junit4:junit4]   2> 6106 T622 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42606, sessionid = 0x13abcb0aee90004, negotiated timeout = 10000
[junit4:junit4]   2> 6106 T615 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 6106 T623 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22932293 name:ZooKeeperConnection Watcher:127.0.0.1:42606/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6107 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6108 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 6113 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 6114 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 6115 T555 oasc.Overseer.start Overseer (id=88590609721917444-127.0.0.1:42606_solr-n_0000000000) starting
[junit4:junit4]   2> 6116 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6118 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6120 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6121 T625 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 6121 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6122 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 6122 T624 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 6124 T624 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6124 T624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 6124 T624 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 6125 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90004 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 6127 T615 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 6127 T621 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 6304 T590 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 6623 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 6629 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90003 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 6630 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 6632 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90003 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6633 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6634 T621 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 6634 T613 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0aee90003
[junit4:junit4]   2> 6634 T615 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 6635 T555 oaz.ZooKeeper.close Session: 0x13abcb0aee90003 closed
[junit4:junit4]   2> 6635 T615 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 6635 T610 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58466 which had sessionid 0x13abcb0aee90003
[junit4:junit4]   2> 6636 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42606/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@8720872
[junit4:junit4]   2> 6637 T626 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42606
[junit4:junit4]   2> 6637 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6638 T626 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42606, initiating session
[junit4:junit4]   2> 6639 T610 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58468
[junit4:junit4]   2> 6639 T610 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58468
[junit4:junit4]   2> 6640 T612 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0aee90005 with negotiated timeout 10000 for client /127.0.0.1:58468
[junit4:junit4]   2> 6640 T626 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42606, sessionid = 0x13abcb0aee90005, negotiated timeout = 10000
[junit4:junit4]   2> 6642 T627 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8720872 name:ZooKeeperConnection Watcher:127.0.0.1:42606/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6642 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6644 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 6646 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 6649 T627 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 6649 T615 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 6649 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6650 T615 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 6653 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 6656 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90005 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 6659 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90005 type:delete cxid:0x14 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard1 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard1
[junit4:junit4]   2> 6660 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 6663 T613 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0aee90005 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6937 T600 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 6938 T600 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 6939 T600 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 6939 T607 oasc.Overseer$ClusterStateUpdater.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 7130 T624 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7132 T624 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 7135 T627 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 7136 T615 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 7136 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7137 T613 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0aee90004
[junit4:junit4]   2> 7138 T623 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 7138 T555 oaz.ZooKeeper.close Session: 0x13abcb0aee90004 closed
[junit4:junit4]   2> 7138 T610 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58467 which had sessionid 0x13abcb0aee90004
[junit4:junit4]   2> 7140 T627 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 7140 T613 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0aee90005
[junit4:junit4]   2> 7140 T615 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 7141 T555 oaz.ZooKeeper.close Session: 0x13abcb0aee90005 closed
[junit4:junit4]   2> 7141 T615 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 7141 T610 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58468 which had sessionid 0x13abcb0aee90005
[junit4:junit4]   2> 7142 T613 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0aee90000
[junit4:junit4]   2> 7143 T555 oaz.ZooKeeper.close Session: 0x13abcb0aee90000 closed
[junit4:junit4]   2> 7143 T610 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58463 which had sessionid 0x13abcb0aee90000
[junit4:junit4]   2> 7143 T613 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 7143 T612 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 7144 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 7146 T610 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 7147 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 7147 T555 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
[junit4:junit4]   2> 7154 T555 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4:junit4]   2> 7155 T555 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 7155 T628 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 7156 T628 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 7157 T628 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351785420697/zookeeper/server1/data/version-2/snapshot.ce
[junit4:junit4]   2> 7161 T628 oazsp.FileTxnSnapLog.save Snapshotting: 11b
[junit4:junit4]   2> 7255 T555 oasc.ZkTestServer.run start zk server on port:48153
[junit4:junit4]   2> 7256 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48153 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6ef06ef0
[junit4:junit4]   2> 7258 T633 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:48153
[junit4:junit4]   2> 7258 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7259 T633 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48153, initiating session
[junit4:junit4]   2> 7259 T629 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46704
[junit4:junit4]   2> 7259 T629 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46704
[junit4:junit4]   2> 7260 T631 oazsp.FileTxnLog.append Creating new log file: log.11c
[junit4:junit4]   2> 7269 T631 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0b3920000 with negotiated timeout 10000 for client /127.0.0.1:46704
[junit4:junit4]   2> 7269 T633 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:48153, sessionid = 0x13abcb0b3920000, negotiated timeout = 10000
[junit4:junit4]   2> 7271 T634 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ef06ef0 name:ZooKeeperConnection Watcher:127.0.0.1:48153 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7271 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7287 T632 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0b3920000
[junit4:junit4]   2> 7287 T634 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 7287 T555 oaz.ZooKeeper.close Session: 0x13abcb0b3920000 closed
[junit4:junit4]   2> 7288 T629 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46704 which had sessionid 0x13abcb0b3920000
[junit4:junit4]   2> 7288 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48153 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@20392039
[junit4:junit4]   2> 7290 T635 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:48153
[junit4:junit4]   2> 7290 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7290 T635 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48153, initiating session
[junit4:junit4]   2> 7291 T629 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46705
[junit4:junit4]   2> 7291 T629 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46705
[junit4:junit4]   2> 7292 T631 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0b3920001 with negotiated timeout 10000 for client /127.0.0.1:46705
[junit4:junit4]   2> 7292 T635 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:48153, sessionid = 0x13abcb0b3920001, negotiated timeout = 10000
[junit4:junit4]   2> 7293 T636 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20392039 name:ZooKeeperConnection Watcher:127.0.0.1:48153 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7294 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7294 T555 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 7295 T632 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0b3920001
[junit4:junit4]   2> 7296 T555 oaz.ZooKeeper.close Session: 0x13abcb0b3920001 closed
[junit4:junit4]   2> 7296 T629 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46705 which had sessionid 0x13abcb0b3920001
[junit4:junit4]   2> 7297 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48153/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@7ec07ec
[junit4:junit4]   2> 7297 T636 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 7298 T637 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:48153
[junit4:junit4]   2> 7298 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7299 T637 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48153, initiating session
[junit4:junit4]   2> 7299 T629 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46706
[junit4:junit4]   2> 7330 T611 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 7331 T629 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46706
[junit4:junit4]   2> 7332 T631 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0b3920002 with negotiated timeout 10000 for client /127.0.0.1:46706
[junit4:junit4]   2> 7333 T637 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:48153, sessionid = 0x13abcb0b3920002, negotiated timeout = 10000
[junit4:junit4]   2> 7333 T638 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ec07ec name:ZooKeeperConnection Watcher:127.0.0.1:48153/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7334 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7334 T555 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 7336 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48153/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@b100b1
[junit4:junit4]   2> 7336 T639 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:48153
[junit4:junit4]   2> 7336 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7338 T639 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48153, initiating session
[junit4:junit4]   2> 7338 T629 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46707
[junit4:junit4]   2> 7338 T629 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46707
[junit4:junit4]   2> 7339 T631 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0b3920003 with negotiated timeout 10000 for client /127.0.0.1:46707
[junit4:junit4]   2> 7339 T639 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:48153, sessionid = 0x13abcb0b3920003, negotiated timeout = 10000
[junit4:junit4]   2> 7340 T640 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b100b1 name:ZooKeeperConnection Watcher:127.0.0.1:48153/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7341 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7342 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 7344 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 7345 T555 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 7346 T555 oasc.Overseer.start Overseer (id=88590609800101891-127.0.0.1:48153_solr-n_0000000000) starting
[junit4:junit4]   2> 7347 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7348 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7349 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7349 T642 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 7350 T641 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 7350 T555 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 7351 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 7352 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:48153/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@686d686d
[junit4:junit4]   2> 7353 T643 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:48153
[junit4:junit4]   2> 7353 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7354 T643 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:48153, initiating session
[junit4:junit4]   2> 7354 T629 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46708
[junit4:junit4]   2> 7355 T629 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46708
[junit4:junit4]   2> 7356 T631 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0b3920004 with negotiated timeout 10000 for client /127.0.0.1:46708
[junit4:junit4]   2> 7356 T643 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:48153, sessionid = 0x13abcb0b3920004, negotiated timeout = 10000
[junit4:junit4]   2> 7357 T644 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@686d686d name:ZooKeeperConnection Watcher:127.0.0.1:48153/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7357 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7358 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 7358 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4:junit4]   2> 7360 T644 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 7360 T638 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 7361 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7362 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 7634 T621 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 7635 T621 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 7635 T621 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 7853 T641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7854 T641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 7855 T641 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4]   2> 7856 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920003 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 7860 T644 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 7860 T638 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 7864 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7868 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7869 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7870 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7872 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8137 T624 oasc.Overseer$ClusterStateUpdater.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 8140 T627 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 8141 T627 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 8142 T615 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 8142 T627 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8143 T615 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8362 T641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8365 T641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core2"}
[junit4:junit4]   2> 8367 T638 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 8367 T644 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 8373 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 8379 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:delete cxid:0x35 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 8380 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 8382 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x3c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8384 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x3e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8870 T641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8873 T641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core3"}
[junit4:junit4]   2> 8875 T638 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 8876 T644 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 8885 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 8888 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:delete cxid:0x4c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard3 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard3
[junit4:junit4]   2> 8889 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4]   2> 8891 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x53 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8908 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x55 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 9378 T641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9419 T641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core4"}
[junit4:junit4]   2> 9479 T638 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 9479 T644 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 9910 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 9911 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x5e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 9914 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x62 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 10020 T641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10021 T641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core5"}
[junit4:junit4]   2> 10040 T638 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 10040 T644 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 10416 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 10418 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x6b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard2/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 10459 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x6f zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 10543 T641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10543 T641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core6"}
[junit4:junit4]   2> 10599 T638 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 10599 T644 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 10999 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 11001 T632 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0b3920004 type:create cxid:0x78 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard3/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 11040 T632 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0b3920002
[junit4:junit4]   2> 11059 T638 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 11058 T555 oaz.ZooKeeper.close Session: 0x13abcb0b3920002 closed
[junit4:junit4]   2> 11059 T629 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46706 which had sessionid 0x13abcb0b3920002
[junit4:junit4]   2> 11079 T644 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 11079 T632 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0b3920004
[junit4:junit4]   2> 11099 T555 oaz.ZooKeeper.close Session: 0x13abcb0b3920004 closed
[junit4:junit4]   2> 11099 T629 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46708 which had sessionid 0x13abcb0b3920004
[junit4:junit4]   2> 11101 T632 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0b3920003
[junit4:junit4]   2> 11119 T640 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 11119 T555 oaz.ZooKeeper.close Session: 0x13abcb0b3920003 closed
[junit4:junit4]   2> 11119 T641 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:738)
[junit4:junit4]   2> 
[junit4:junit4]   2> 11119 T629 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46707 which had sessionid 0x13abcb0b3920003
[junit4:junit4]   2> 11120 T632 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 11120 T631 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 11120 T641 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88590609800101891-127.0.0.1:48153_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 11122 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 11139 T629 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 11140 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 11141 T555 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4:junit4]   2> 11154 T555 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
[junit4:junit4]   2> 11155 T555 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 11157 T645 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 11158 T645 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 11161 T645 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351785420697/zookeeper/server1/data/version-2/snapshot.11b
[junit4:junit4]   2> 11169 T645 oazsp.FileTxnSnapLog.save Snapshotting: 194
[junit4:junit4]   2> 11256 T555 oasc.ZkTestServer.run start zk server on port:42924
[junit4:junit4]   2> 11258 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42924/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@64876487
[junit4:junit4]   2> 11259 T650 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42924
[junit4:junit4]   2> 11259 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 11261 T650 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42924, initiating session
[junit4:junit4]   2> 11261 T646 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60049
[junit4:junit4]   2> 11263 T646 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60049
[junit4:junit4]   2> 11264 T648 oazsp.FileTxnLog.append Creating new log file: log.195
[junit4:junit4]   2> 11279 T648 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0c33b0000 with negotiated timeout 10000 for client /127.0.0.1:60049
[junit4:junit4]   2> 11280 T650 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42924, sessionid = 0x13abcb0c33b0000, negotiated timeout = 10000
[junit4:junit4]   2> 11281 T651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64876487 name:ZooKeeperConnection Watcher:127.0.0.1:42924/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 11281 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 11282 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42924 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@5a245a24
[junit4:junit4]   2> 11283 T652 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42924
[junit4:junit4]   2> 11283 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 11284 T652 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42924, initiating session
[junit4:junit4]   2> 11284 T646 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60050
[junit4:junit4]   2> 11285 T646 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60050
[junit4:junit4]   2> 11286 T652 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42924, sessionid = 0x13abcb0c33b0001, negotiated timeout = 10000
[junit4:junit4]   2> 11286 T648 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0c33b0001 with negotiated timeout 10000 for client /127.0.0.1:60050
[junit4:junit4]   2> 11286 T653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a245a24 name:ZooKeeperConnection Watcher:127.0.0.1:42924 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 11287 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 11304 T630 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 11310 T649 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0c33b0001
[junit4:junit4]   2> 11319 T653 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 11319 T555 oaz.ZooKeeper.close Session: 0x13abcb0c33b0001 closed
[junit4:junit4]   2> 11320 T646 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60050 which had sessionid 0x13abcb0c33b0001
[junit4:junit4]   2> 11321 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42924 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@46ad46ad
[junit4:junit4]   2> 11322 T654 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42924
[junit4:junit4]   2> 11322 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 11323 T654 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42924, initiating session
[junit4:junit4]   2> 11323 T646 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60051
[junit4:junit4]   2> 11324 T646 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60051
[junit4:junit4]   2> 11326 T648 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0c33b0002 with negotiated timeout 10000 for client /127.0.0.1:60051
[junit4:junit4]   2> 11326 T654 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42924, sessionid = 0x13abcb0c33b0002, negotiated timeout = 10000
[junit4:junit4]   2> 11327 T655 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46ad46ad name:ZooKeeperConnection Watcher:127.0.0.1:42924 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 11328 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 11329 T555 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 11330 T649 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0c33b0002
[junit4:junit4]   2> 11331 T655 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 11331 T555 oaz.ZooKeeper.close Session: 0x13abcb0c33b0002 closed
[junit4:junit4]   2> 11331 T646 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60051 which had sessionid 0x13abcb0c33b0002
[junit4:junit4]   2> 11332 T555 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 11334 T656 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42924/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4ec64ec6
[junit4:junit4]   2> 11334 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42924/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@20882088
[junit4:junit4]   2> 11335 T657 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42924
[junit4:junit4]   2> 11335 T656 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 11336 T659 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42924
[junit4:junit4]   2> 11336 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 11336 T657 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42924, initiating session
[junit4:junit4]   2> 11337 T659 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42924, initiating session
[junit4:junit4]   2> 11336 T646 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60052
[junit4:junit4]   2> 11338 T646 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60052
[junit4:junit4]   2> 11339 T646 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60053
[junit4:junit4]   2> 11339 T646 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60053
[junit4:junit4]   2> 11339 T648 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0c33b0003 with negotiated timeout 10000 for client /127.0.0.1:60052
[junit4:junit4]   2> 11339 T657 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42924, sessionid = 0x13abcb0c33b0003, negotiated timeout = 10000
[junit4:junit4]   2> 11340 T658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ec64ec6 name:ZooKeeperConnection Watcher:127.0.0.1:42924/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 11340 T648 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0c33b0004 with negotiated timeout 10000 for client /127.0.0.1:60053
[junit4:junit4]   2> 11341 T659 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42924, sessionid = 0x13abcb0c33b0004, negotiated timeout = 10000
[junit4:junit4]   2> 11341 T656 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 11342 T660 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20882088 name:ZooKeeperConnection Watcher:127.0.0.1:42924/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 11342 T656 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 11343 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 11344 T555 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 11345 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 11345 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 11346 T656 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 11346 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 11347 T656 oasc.Overseer.start Overseer (id=88590610062835715-127.0.0.1:42924_solr-n_0000000000) starting
[junit4:junit4]   2> 11347 T660 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 11347 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0004 type:create cxid:0xd zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11348 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11349 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0004 type:create cxid:0xe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 11349 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11350 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11351 T662 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 11351 T656 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4:junit4]   2> 11352 T649 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0c33b0003
[junit4:junit4]   2> 11353 T658 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 11353 T656 oaz.ZooKeeper.close Session: 0x13abcb0c33b0003 closed
[junit4:junit4]   2> 11353 T646 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60052 which had sessionid 0x13abcb0c33b0003
[junit4:junit4]   2> 11353 T656 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42924/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@10f010f0
[junit4:junit4]   2> 11354 T663 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42924
[junit4:junit4]   2> 11354 T656 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 11356 T663 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42924, initiating session
[junit4:junit4]   2> 11356 T646 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60054
[junit4:junit4]   2> 11356 T646 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60054
[junit4:junit4]   2> 11357 T648 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0c33b0005 with negotiated timeout 10000 for client /127.0.0.1:60054
[junit4:junit4]   2> 11357 T663 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42924, sessionid = 0x13abcb0c33b0005, negotiated timeout = 10000
[junit4:junit4]   2> 11367 T664 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10f010f0 name:ZooKeeperConnection Watcher:127.0.0.1:42924/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 11368 T656 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 11370 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 11371 T656 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 11372 T656 oasc.Overseer.start Overseer (id=88590610062835717-127.0.0.1:42924_solr-n_0000000002) starting
[junit4:junit4]   2> 11372 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0005 type:create cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11388 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11389 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0005 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11390 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0005 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11391 T666 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 11391 T665 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 11393 T665 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11393 T665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"state1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 11394 T665 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 11394 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0005 type:create cxid:0x18 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 11398 T660 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 11851 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 11891 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0004 type:delete cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 11891 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 11893 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0004 type:create cxid:0x28 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11894 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0004 type:create cxid:0x2a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11895 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 11896 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0004 type:create cxid:0x31 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 11901 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42924/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@29132913
[junit4:junit4]   2> 11902 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 11902 T667 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42924
[junit4:junit4]   2> 11902 T665 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11904 T667 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42924, initiating session
[junit4:junit4]   2> 11904 T646 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60056
[junit4:junit4]   2> 11905 T646 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60056
[junit4:junit4]   2> 11906 T648 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb0c33b0006 with negotiated timeout 10000 for client /127.0.0.1:60056
[junit4:junit4]   2> 11906 T667 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42924, sessionid = 0x13abcb0c33b0006, negotiated timeout = 10000
[junit4:junit4]   2> 11907 T668 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@29132913 name:ZooKeeperConnection Watcher:127.0.0.1:42924/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 11908 T665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"state2",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 11908 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 11909 T555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 11935 T660 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 11935 T668 oaz.ClientCnxn$EventThread.processEvent SEVERE Error while calling watcher  java.lang.NullPointerException
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:189)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 11936 T555 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4:junit4]   2> 11945 T660 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 11945 T668 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 11946 T660 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 11946 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0004 type:create cxid:0x3a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11948 T555 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 11949 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0004 type:create cxid:0x41 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 11951 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11951 T649 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13abcb0c33b0006 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12080 T644 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 12081 T644 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 12081 T644 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 12082 T644 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 12083 T644 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 12084 T644 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12352 T661 oasc.Overseer$ClusterStateUpdater.run WARNING Solr cannot talk to ZK
[junit4:junit4]   2> 12352 T662 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 12437 T665 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12438 T665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"state1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 12441 T665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"state2",
[junit4:junit4]   2> 	  "base_url":"http://node2/solr/",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "node_name":"node2",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core4"}
[junit4:junit4]   2> 12444 T660 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 71975 T660 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 71975 T649 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0c33b0004
[junit4:junit4]   2> 71975 T668 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 71976 T555 oaz.ZooKeeper.close Session: 0x13abcb0c33b0004 closed
[junit4:junit4]   2> 71976 T646 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60053 which had sessionid 0x13abcb0c33b0004
[junit4:junit4]   2> 71977 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72078 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72181 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72283 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72384 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72485 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72587 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72689 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72791 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72892 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72975 T660 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 72976 T660 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 72977 T660 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 72977 T660 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 72978 T660 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 72994 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73095 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73197 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73298 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73400 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73501 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73603 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73704 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73805 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73907 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74008 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74111 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74212 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74314 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74415 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74518 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74619 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74721 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74822 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74924 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75025 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75127 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75238 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75340 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75441 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75542 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75644 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75745 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75846 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75947 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76049 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76157 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76262 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76364 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76465 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76566 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76668 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76774 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76876 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76977 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77079 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77180 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77281 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77383 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77484 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77586 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77688 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77789 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77891 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77992 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78094 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78196 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78297 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78399 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78500 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78602 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78703 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78805 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78906 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79007 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79109 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79210 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79312 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79413 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79521 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79625 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79726 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79828 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79929 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80030 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80135 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80236 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80338 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80439 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80540 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80642 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80743 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80844 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80946 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81047 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81149 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81250 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81352 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81453 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81555 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81656 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81758 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81860 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81961 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82063 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82165 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82266 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82381 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82482 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82584 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82697 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82801 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82917 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83019 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83120 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83222 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83323 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83424 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83526 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83627 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83729 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83830 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83932 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84033 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84135 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84237 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84338 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84440 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84541 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84643 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84744 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84845 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84947 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85048 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85150 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85257 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85361 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85462 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85569 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85670 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85772 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85873 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85975 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86076 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86178 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86280 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86381 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86483 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86584 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86685 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86788 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86889 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86990 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87092 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87193 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87295 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87396 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87498 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87599 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87705 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87806 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87908 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88010 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88111 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88212 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88314 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88425 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88526 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88628 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88729 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88831 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88932 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89033 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89134 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89236 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89338 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89439 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89541 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89642 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89743 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89845 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89946 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90047 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90149 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90250 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90352 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90457 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90561 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90665 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90767 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90868 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90970 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91071 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91173 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91274 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91376 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91477 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91578 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91680 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91781 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91883 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 91984 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 92085 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 92187 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 92288 T555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 92395 T649 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0c33b0005
[junit4:junit4]   2> 92396 T656 oaz.ZooKeeper.close Session: 0x13abcb0c33b0005 closed
[junit4:junit4]   2> 92396 T664 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 92396 T646 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60054 which had sessionid 0x13abcb0c33b0005
[junit4:junit4]   2> 92398 T649 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0c33b0006
[junit4:junit4]   2> 92398 T668 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 92399 T555 oaz.ZooKeeper.close Session: 0x13abcb0c33b0006 closed
[junit4:junit4]   2> 92399 T668 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 92399 T646 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60056 which had sessionid 0x13abcb0c33b0006
[junit4:junit4]   2> 92401 T649 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb0c33b0000
[junit4:junit4]   2> 92401 T651 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 92401 T555 oaz.ZooKeeper.close Session: 0x13abcb0c33b0000 closed
[junit4:junit4]   2> 92402 T646 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60049 which had sessionid 0x13abcb0c33b0000
[junit4:junit4]   2> 92402 T649 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 92402 T648 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 92404 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 92405 T646 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 92406 T555 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 92407 T555 oas.SolrTestCaseJ4.tearDown ###Ending testShardLeaderChange
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testShardLeaderChange -Dtests.seed=DA6A5DCD24EF6E3B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_GB -Dtests.timezone=Etc/GMT+8 -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 81.3s J1 | OverseerTest.testShardLeaderChange <<<
[junit4:junit4]    > Throwable #1: org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([DA6A5DCD24EF6E3B:439DA3A3E779BCA]:0)
[junit4:junit4]    > 	at org.junit.Assert.assertEquals(Assert.java:125)
[junit4:junit4]    > 	at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:493)
[junit4:junit4]    > 	at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:652)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:611)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:738)
[junit4:junit4]   2> 92436 T555 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4:junit4]   2> 92437 T555 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 92438 T669 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 92438 T669 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 92439 T669 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351785420697/zookeeper/server1/data/version-2/snapshot.194
[junit4:junit4]   2> 92445 T669 oazsp.FileTxnSnapLog.save Snapshotting: 1fa
[junit4:junit4]   2> 92537 T555 oasc.ZkTestServer.run start zk server on port:54763
[junit4:junit4]   2> 92538 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:54763/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@61746174
[junit4:junit4]   2> 92539 T674 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:54763
[junit4:junit4]   2> 92539 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 92540 T674 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:54763, initiating session
[junit4:junit4]   2> 92540 T670 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42542
[junit4:junit4]   2> 92541 T670 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42542
[junit4:junit4]   2> 92542 T672 oazsp.FileTxnLog.append Creating new log file: log.1fb
[junit4:junit4]   2> 92545 T672 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb200b60000 with negotiated timeout 10000 for client /127.0.0.1:42542
[junit4:junit4]   2> 92545 T674 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:54763, sessionid = 0x13abcb200b60000, negotiated timeout = 10000
[junit4:junit4]   2> 92546 T675 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61746174 name:ZooKeeperConnection Watcher:127.0.0.1:54763/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 92546 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 92547 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:54763 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@58f458f4
[junit4:junit4]   2> 92548 T676 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:54763
[junit4:junit4]   2> 92548 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 92549 T676 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:54763, initiating session
[junit4:junit4]   2> 92549 T670 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42543
[junit4:junit4]   2> 92550 T670 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42543
[junit4:junit4]   2> 92552 T672 oazs.NIOServerCnxn.finishSessionInit Established session 0x13abcb200b60001 with negotiated timeout 10000 for client /127.0.0.1:42543
[junit4:junit4]   2> 92552 T676 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:54763, sessionid = 0x13abcb200b60001, negotiated timeout = 10000
[junit4:junit4]   2> 92553 T677 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58f458f4 name:ZooKeeperConnection Watcher:127.0.0.1:54763 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 92554 T555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 92565 T673 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13abcb200b60001
[junit4:junit4]   2> 92566 T670 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:42543 which had sessionid 0x13abcb200b60001
[junit4:junit4]   2> 92567 T555 oaz.ZooKeeper.close Session: 0x13abcb200b60001 closed
[junit4:junit4]   2> 92568 T555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:54763 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@22c922c9
[junit4:junit4]   2> 92569 T678 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:54763
[junit4:junit4]   2> 92569 T555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 92569 T677 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 92603 T670 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42546
[junit4:junit4]   2> 92603 T678 oaz.ClientCnxn$SendThread.primeConnection Socket connection established t

[...truncated too long message...]

[junit4:junit4]   2> 103082 T792 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103083 T794 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103084 T794 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103084 T794 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103094 T796 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103094 T796 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103121 T798 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leaders
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:176)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:173)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:173)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:384)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:353)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:340)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:95)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:155)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:99)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 103121 T798 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103122 T798 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (8)
[junit4:junit4]   2> 103122 T798 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103122 T798 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103130 T800 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103131 T800 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103131 T800 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103139 T802 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103139 T802 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103140 T802 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103158 T808 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:643)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:210)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:207)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:207)
[junit4:junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:226)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:104)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:155)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:99)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 103159 T808 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103160 T808 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 103160 T808 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103160 T808 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103161 T810 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leaders/shard18
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:643)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:399)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:396)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:353)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:340)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:95)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:155)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:99)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 103162 T810 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 103162 T810 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103162 T810 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103163 T810 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 105184 T555 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=en_GB, timezone=Etc/GMT+8
[junit4:junit4]   2> NOTE: Linux 3.2.0-32-generic amd64/IBM Corporation 1.6.0 (64-bit)/cpus=8,threads=10,free=13855224,total=113158144
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestRemoteStreaming, TestStressLucene, TestSolrCoreProperties, ZkCLITest, IndexSchemaTest, PingRequestHandlerTest, LeaderElectionTest, SpellingQueryConverterTest, BasicDistributedZk2Test, TestValueSourceCache, IndexSchemaRuntimeFieldTest, CoreContainerCoreInitFailuresTest, SignatureUpdateProcessorFactoryTest, OverseerTest]
[junit4:junit4] Completed on J1 in 105.23s, 8 tests, 1 failure <<< FAILURES!

[...truncated 667 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:178: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1163: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 1 failure, 7 ignored (2 assumptions)

Total time: 31 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 64bit/ibm-j9-jdk6 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message