lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@sd-datasolutions.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.6.0_37) - Build # 2282 - Failure!
Date Thu, 08 Nov 2012 18:14:12 GMT
Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Linux/2282/
Java: 32bit/jdk1.6.0_37 -server -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testShardLeaderChange

Error Message:
Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>

Stack Trace:
org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>
	at __randomizedtesting.SeedInfo.seed([6192F3A8BF7966FC:BFC1745FA5E1930D]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:493)
	at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:652)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:662)




Build Log:
[...truncated 8113 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4:junit4]   2> 0 T1143 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1352398067346
[junit4:junit4]   2> 0 T1143 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 3 T1143 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
[junit4:junit4]   2> 3 T1143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 4 T1144 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 4 T1144 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 5 T1144 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 103 T1143 oasc.ZkTestServer.run start zk server on port:47207
[junit4:junit4]   2> 104 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47207/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@8c296d
[junit4:junit4]   2> 105 T1149 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47207
[junit4:junit4]   2> 105 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 105 T1149 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47207, initiating session
[junit4:junit4]   2> 106 T1145 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:54925
[junit4:junit4]   2> 106 T1145 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:54925
[junit4:junit4]   2> 106 T1147 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 212 T1147 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134d6970000 with negotiated timeout 10000 for client /127.0.0.1:54925
[junit4:junit4]   2> 212 T1149 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47207, sessionid = 0x13ae134d6970000, negotiated timeout = 10000
[junit4:junit4]   2> 213 T1150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8c296d name:ZooKeeperConnection Watcher:127.0.0.1:47207/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 213 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 213 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47207 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1f329ac
[junit4:junit4]   2> 214 T1151 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47207
[junit4:junit4]   2> 214 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 215 T1151 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47207, initiating session
[junit4:junit4]   2> 215 T1145 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:54926
[junit4:junit4]   2> 215 T1145 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:54926
[junit4:junit4]   2> 216 T1147 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134d6970001 with negotiated timeout 10000 for client /127.0.0.1:54926
[junit4:junit4]   2> 216 T1151 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47207, sessionid = 0x13ae134d6970001, negotiated timeout = 10000
[junit4:junit4]   2> 216 T1152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f329ac name:ZooKeeperConnection Watcher:127.0.0.1:47207 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 216 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 217 T1148 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134d6970001
[junit4:junit4]   2> 217 T1152 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 217 T1145 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:54926 which had sessionid 0x13ae134d6970001
[junit4:junit4]   2> 217 T1143 oaz.ZooKeeper.close Session: 0x13ae134d6970001 closed
[junit4:junit4]   2> 218 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47207 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1558803
[junit4:junit4]   2> 219 T1153 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47207
[junit4:junit4]   2> 219 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 219 T1145 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:54927
[junit4:junit4]   2> 219 T1153 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47207, initiating session
[junit4:junit4]   2> 219 T1145 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:54927
[junit4:junit4]   2> 220 T1147 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134d6970002 with negotiated timeout 10000 for client /127.0.0.1:54927
[junit4:junit4]   2> 220 T1153 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47207, sessionid = 0x13ae134d6970002, negotiated timeout = 10000
[junit4:junit4]   2> 220 T1154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1558803 name:ZooKeeperConnection Watcher:127.0.0.1:47207 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 220 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 221 T1143 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 222 T1148 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134d6970002
[junit4:junit4]   2> 223 T1154 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 223 T1145 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:54927 which had sessionid 0x13ae134d6970002
[junit4:junit4]   2> 223 T1143 oaz.ZooKeeper.close Session: 0x13ae134d6970002 closed
[junit4:junit4]   2> 223 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 225 T1143 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 226 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 228 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47207/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@3039ce
[junit4:junit4]   2> 228 T1155 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47207
[junit4:junit4]   2> 228 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 229 T1155 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47207, initiating session
[junit4:junit4]   2> 229 T1145 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:54928
[junit4:junit4]   2> 229 T1145 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:54928
[junit4:junit4]   2> 230 T1147 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134d6970003 with negotiated timeout 10000 for client /127.0.0.1:54928
[junit4:junit4]   2> 230 T1155 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47207, sessionid = 0x13ae134d6970003, negotiated timeout = 10000
[junit4:junit4]   2> 230 T1156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3039ce name:ZooKeeperConnection Watcher:127.0.0.1:47207/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 230 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 231 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 233 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 235 T1150 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 238 T1156 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 238 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47207/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@729834
[junit4:junit4]   2> 239 T1157 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47207
[junit4:junit4]   2> 239 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 239 T1157 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47207, initiating session
[junit4:junit4]   2> 239 T1145 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:54929
[junit4:junit4]   2> 240 T1145 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:54929
[junit4:junit4]   2> 240 T1157 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47207, sessionid = 0x13ae134d6970004, negotiated timeout = 10000
[junit4:junit4]   2> 240 T1147 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134d6970004 with negotiated timeout 10000 for client /127.0.0.1:54929
[junit4:junit4]   2> 240 T1158 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@729834 name:ZooKeeperConnection Watcher:127.0.0.1:47207/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 241 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 242 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 245 T1148 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134d6970004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 245 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 247 T1143 oasc.Overseer.start Overseer (id=88630759741915140-127.0.0.1:47207_solr-n_0000000000) starting
[junit4:junit4]   2> 248 T1148 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134d6970004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 249 T1148 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134d6970004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 249 T1148 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134d6970004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 250 T1160 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 250 T1148 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134d6970003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 251 T1159 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 251 T1148 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134d6970003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 754 T1159 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 754 T1159 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"12",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 754 T1159 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 12
[junit4:junit4]   2> 755 T1148 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134d6970004 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 758 T1150 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 758 T1156 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 1252 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 1259 T1148 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134d6970003 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 1259 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 1262 T1148 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134d6970003 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1263 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1264 T1148 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134d6970004
[junit4:junit4]   2> 1264 T1158 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1264 T1145 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:54929 which had sessionid 0x13ae134d6970004
[junit4:junit4]   2> 1264 T1143 oaz.ZooKeeper.close Session: 0x13ae134d6970004 closed
[junit4:junit4]   2> 1265 T1148 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134d6970003
[junit4:junit4]   2> 1265 T1150 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 1266 T1156 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 1266 T1150 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 1266 T1156 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 1266 T1145 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:54928 which had sessionid 0x13ae134d6970003
[junit4:junit4]   2> 1266 T1143 oaz.ZooKeeper.close Session: 0x13ae134d6970003 closed
[junit4:junit4]   2> 1267 T1148 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134d6970000
[junit4:junit4]   2> 1267 T1150 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1267 T1145 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:54925 which had sessionid 0x13ae134d6970000
[junit4:junit4]   2> 1267 T1143 oaz.ZooKeeper.close Session: 0x13ae134d6970000 closed
[junit4:junit4]   2> 1267 T1148 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 1267 T1147 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 1268 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 1269 T1145 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 1269 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 1269 T1143 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
[junit4:junit4]   2> 1273 T1143 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
[junit4:junit4]   2> 1274 T1143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 1274 T1161 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 1274 T1161 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 1275 T1161 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 1276 T1161 oazsp.FileTxnSnapLog.save Snapshotting: 2e
[junit4:junit4]   2> 1374 T1143 oasc.ZkTestServer.run start zk server on port:50631
[junit4:junit4]   2> 1375 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50631/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@cf0b27
[junit4:junit4]   2> 1375 T1166 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50631
[junit4:junit4]   2> 1375 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1376 T1166 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50631, initiating session
[junit4:junit4]   2> 1376 T1162 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:45364
[junit4:junit4]   2> 1376 T1162 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:45364
[junit4:junit4]   2> 1376 T1164 oazsp.FileTxnLog.append Creating new log file: log.2f
[junit4:junit4]   2> 1378 T1164 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134db8f0000 with negotiated timeout 10000 for client /127.0.0.1:45364
[junit4:junit4]   2> 1378 T1166 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50631, sessionid = 0x13ae134db8f0000, negotiated timeout = 10000
[junit4:junit4]   2> 1378 T1167 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cf0b27 name:ZooKeeperConnection Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1378 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1378 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50631 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@11b3588
[junit4:junit4]   2> 1379 T1168 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50631
[junit4:junit4]   2> 1379 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1379 T1168 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50631, initiating session
[junit4:junit4]   2> 1379 T1162 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:45365
[junit4:junit4]   2> 1380 T1162 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:45365
[junit4:junit4]   2> 1380 T1164 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134db8f0001 with negotiated timeout 10000 for client /127.0.0.1:45365
[junit4:junit4]   2> 1380 T1168 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50631, sessionid = 0x13ae134db8f0001, negotiated timeout = 10000
[junit4:junit4]   2> 1380 T1169 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11b3588 name:ZooKeeperConnection Watcher:127.0.0.1:50631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1381 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1389 T1165 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134db8f0001
[junit4:junit4]   2> 1390 T1169 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1390 T1162 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45365 which had sessionid 0x13ae134db8f0001
[junit4:junit4]   2> 1390 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f0001 closed
[junit4:junit4]   2> 1391 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50631 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@34c129
[junit4:junit4]   2> 1391 T1170 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50631
[junit4:junit4]   2> 1391 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1391 T1170 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50631, initiating session
[junit4:junit4]   2> 1392 T1162 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:45366
[junit4:junit4]   2> 1392 T1162 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:45366
[junit4:junit4]   2> 1392 T1164 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134db8f0002 with negotiated timeout 10000 for client /127.0.0.1:45366
[junit4:junit4]   2> 1392 T1170 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50631, sessionid = 0x13ae134db8f0002, negotiated timeout = 10000
[junit4:junit4]   2> 1393 T1171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34c129 name:ZooKeeperConnection Watcher:127.0.0.1:50631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1393 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1393 T1143 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 1394 T1165 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134db8f0002
[junit4:junit4]   2> 1395 T1171 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1395 T1162 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45366 which had sessionid 0x13ae134db8f0002
[junit4:junit4]   2> 1395 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f0002 closed
[junit4:junit4]   2> 1395 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1397 T1172 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50631/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1333f77
[junit4:junit4]   2> 1397 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50631/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@18bd49c
[junit4:junit4]   2> 1397 T1173 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50631
[junit4:junit4]   2> 1397 T1172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1404 T1175 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50631
[junit4:junit4]   2> 1404 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1405 T1162 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:45367
[junit4:junit4]   2> 1405 T1173 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50631, initiating session
[junit4:junit4]   2> 1405 T1175 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50631, initiating session
[junit4:junit4]   2> 1405 T1162 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:45368
[junit4:junit4]   2> 1406 T1162 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:45367
[junit4:junit4]   2> 1406 T1162 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:45368
[junit4:junit4]   2> 1406 T1164 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134db8f0003 with negotiated timeout 10000 for client /127.0.0.1:45367
[junit4:junit4]   2> 1407 T1173 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50631, sessionid = 0x13ae134db8f0003, negotiated timeout = 10000
[junit4:junit4]   2> 1407 T1174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1333f77 name:ZooKeeperConnection Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1407 T1172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1407 T1175 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50631, sessionid = 0x13ae134db8f0004, negotiated timeout = 10000
[junit4:junit4]   2> 1407 T1164 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134db8f0004 with negotiated timeout 10000 for client /127.0.0.1:45368
[junit4:junit4]   2> 1407 T1176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18bd49c name:ZooKeeperConnection Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1408 T1172 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1408 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1408 T1143 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1410 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1410 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1410 T1172 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1411 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 1412 T1172 oasc.Overseer.start Overseer (id=88630759825276931-127.0.0.1:50631_solr-n_0000000000) starting
[junit4:junit4]   2> 1414 T1176 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 1414 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1414 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0004 type:create cxid:0xd zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1415 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1415 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0004 type:create cxid:0xe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 1415 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1416 T1178 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1416 T1172 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4:junit4]   2> 1416 T1165 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134db8f0003
[junit4:junit4]   2> 1416 T1174 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1417 T1162 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45367 which had sessionid 0x13ae134db8f0003
[junit4:junit4]   2> 1416 T1172 oaz.ZooKeeper.close Session: 0x13ae134db8f0003 closed
[junit4:junit4]   2> 1417 T1172 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50631/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1ee1fda
[junit4:junit4]   2> 1417 T1179 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50631
[junit4:junit4]   2> 1417 T1172 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1418 T1179 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50631, initiating session
[junit4:junit4]   2> 1418 T1162 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:45369
[junit4:junit4]   2> 1418 T1162 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:45369
[junit4:junit4]   2> 1419 T1164 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134db8f0005 with negotiated timeout 10000 for client /127.0.0.1:45369
[junit4:junit4]   2> 1419 T1179 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50631, sessionid = 0x13ae134db8f0005, negotiated timeout = 10000
[junit4:junit4]   2> 1420 T1180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ee1fda name:ZooKeeperConnection Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1420 T1172 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1421 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1421 T1172 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1422 T1172 oasc.Overseer.start Overseer (id=88630759825276933-127.0.0.1:50631_solr-n_0000000002) starting
[junit4:junit4]   2> 1423 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0005 type:create cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1423 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1424 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0005 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1424 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0005 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1425 T1182 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1425 T1181 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1426 T1181 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1426 T1181 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "state":"state1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 1427 T1181 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 1427 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0005 type:create cxid:0x18 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 1430 T1176 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 1655 T1146 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 1760 T1159 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 
[junit4:junit4]   2> 1762 T1159 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88630759741915140-127.0.0.1:47207_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 1917 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 1922 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0004 type:delete cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 1923 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 1928 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0004 type:create cxid:0x28 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1929 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0004 type:create cxid:0x2a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1930 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 1932 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0004 type:create cxid:0x31 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 1932 T1181 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1933 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:50631/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@8d844b
[junit4:junit4]   2> 1934 T1183 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:50631
[junit4:junit4]   2> 1934 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1935 T1183 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:50631, initiating session
[junit4:junit4]   2> 1935 T1162 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:45370
[junit4:junit4]   2> 1935 T1181 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "state":"state2",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 1935 T1162 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:45370
[junit4:junit4]   2> 1936 T1164 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae134db8f0006 with negotiated timeout 10000 for client /127.0.0.1:45370
[junit4:junit4]   2> 1936 T1183 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:50631, sessionid = 0x13ae134db8f0006, negotiated timeout = 10000
[junit4:junit4]   2> 1936 T1184 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8d844b name:ZooKeeperConnection Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1936 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1937 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2218 T1176 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2218 T1184 oaz.ClientCnxn$EventThread.processEvent SEVERE Error while calling watcher  java.lang.NullPointerException
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:189)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 2218 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4:junit4]   2> 2221 T1176 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2222 T1184 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2222 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0004 type:create cxid:0x3a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2222 T1176 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2223 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2224 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0004 type:create cxid:0x41 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2226 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2227 T1165 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae134db8f0006 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2266 T1156 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 2266 T1156 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 2416 T1178 oasc.OverseerCollectionProcessor.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.amILeader(OverseerCollectionProcessor.java:129)
[junit4:junit4]   2> 	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:84)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 
[junit4:junit4]   2> 2416 T1177 oasc.Overseer$ClusterStateUpdater.run WARNING Solr cannot talk to ZK
[junit4:junit4]   2> 2417 T1178 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=88630759825276931-127.0.0.1:50631_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 2723 T1181 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2723 T1181 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "state":"state1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 2725 T1181 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "state":"state2",
[junit4:junit4]   2> 	  "core":"core4",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node2",
[junit4:junit4]   2> 	  "base_url":"http://node2/solr/"}
[junit4:junit4]   2> 2727 T1176 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 62244 T1176 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 62244 T1165 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134db8f0004
[junit4:junit4]   2> 62245 T1184 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 62245 T1162 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45368 which had sessionid 0x13ae134db8f0004
[junit4:junit4]   2> 62245 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f0004 closed
[junit4:junit4]   2> 62246 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62347 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62448 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62550 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62651 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62753 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62855 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62956 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63057 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63159 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63245 T1176 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 63245 T1176 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 63245 T1176 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 63245 T1176 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 63246 T1176 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 63260 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63361 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63463 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63564 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63665 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63767 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63868 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63969 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64071 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64172 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64274 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64376 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64477 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64579 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64680 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64782 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64883 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64984 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65085 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65187 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65288 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65390 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65491 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65592 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65694 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65795 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65897 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65998 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66099 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66201 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66302 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66403 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66505 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66606 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66707 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66809 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66910 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67011 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67113 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67215 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67316 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67418 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67519 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67621 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67722 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67824 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67925 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68026 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68128 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68229 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68331 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68432 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68533 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68635 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68736 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68837 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68938 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69040 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69141 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69243 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69344 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69445 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69547 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69648 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69749 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69851 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69952 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70054 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70156 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70257 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70359 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70460 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70562 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70664 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70765 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70867 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70969 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71070 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71171 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71273 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71374 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71475 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71576 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71678 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71779 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71880 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71982 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72083 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72185 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72286 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72387 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72489 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72590 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72691 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72793 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72894 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72996 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73097 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73198 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73300 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73402 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73503 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73605 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73706 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73808 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73909 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74010 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74112 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74214 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74315 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74417 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74519 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74620 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74722 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74823 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74925 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75026 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75128 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75229 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75330 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75432 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75533 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75634 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75736 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75837 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75939 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76040 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76142 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76243 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76345 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76446 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76547 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76649 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76750 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76852 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76953 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77055 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77156 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77258 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77359 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77460 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77562 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77663 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77764 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77866 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77967 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78068 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78169 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78271 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78372 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78474 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78576 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78678 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78779 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78881 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78982 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79084 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79186 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79287 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79388 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79490 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79592 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79693 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79795 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79896 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79997 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80098 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80203 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80304 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80406 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80507 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80609 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80710 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80811 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80912 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81014 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81115 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81216 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81318 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81419 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81520 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81622 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81723 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81824 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81926 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82027 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82129 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82230 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82332 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82433 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82604 T1165 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134db8f0005
[junit4:junit4]   2> 82605 T1172 oaz.ZooKeeper.close Session: 0x13ae134db8f0005 closed
[junit4:junit4]   2> 82605 T1180 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82605 T1162 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45369 which had sessionid 0x13ae134db8f0005
[junit4:junit4]   2> 82606 T1165 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134db8f0006
[junit4:junit4]   2> 82606 T1184 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 82607 T1184 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82607 T1162 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45370 which had sessionid 0x13ae134db8f0006
[junit4:junit4]   2> 82607 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f0006 closed
[junit4:junit4]   2> 82607 T1165 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae134db8f0000
[junit4:junit4]   2> 82608 T1167 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82608 T1162 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45364 which had sessionid 0x13ae134db8f0000
[junit4:junit4]   2> 82608 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f0000 closed
[junit4:junit4]   2> 82608 T1165 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 82608 T1164 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 82609 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 82611 T1162 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 82611 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 82612 T1143 oas.SolrTestCaseJ4.tearDown ###Ending testShardLeaderChange
[junit4:junit4]   2> 82654 T1163 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testShardLeaderChange -Dtests.seed=6192F3A8BF7966FC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=nl_NL -Dtests.timezone=America/Aruba -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 81.4s J1 | OverseerTest.testShardLeaderChange <<<
[junit4:junit4]    > Throwable #1: org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([6192F3A8BF7966FC:BFC1745FA5E1930D]:0)
[junit4:junit4]    > 	at org.junit.Assert.assertEquals(Assert.java:125)
[junit4:junit4]    > 	at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:493)
[junit4:junit4]    > 	at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:652)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 82663 T1143 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
[junit4:junit4]   2> 82663 T1143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 82664 T1185 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 82664 T1185 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 82665 T1185 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version-2/snapshot.2e
[junit4:junit4]   2> 82668 T1185 oazsp.FileTxnSnapLog.save Snapshotting: 91
[junit4:junit4]   2> 82764 T1143 oasc.ZkTestServer.run start zk server on port:59263
[junit4:junit4]   2> 82764 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59263/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@10d6341
[junit4:junit4]   2> 82765 T1190 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59263
[junit4:junit4]   2> 82765 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 82765 T1190 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59263, initiating session
[junit4:junit4]   2> 82765 T1186 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35653
[junit4:junit4]   2> 82766 T1186 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35653
[junit4:junit4]   2> 82766 T1188 oazsp.FileTxnLog.append Creating new log file: log.92
[junit4:junit4]   2> 82767 T1188 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136197f0000 with negotiated timeout 10000 for client /127.0.0.1:35653
[junit4:junit4]   2> 82767 T1190 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59263, sessionid = 0x13ae136197f0000, negotiated timeout = 10000
[junit4:junit4]   2> 82768 T1191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10d6341 name:ZooKeeperConnection Watcher:127.0.0.1:59263/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 82768 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 82768 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59263 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1ac9a93
[junit4:junit4]   2> 82769 T1192 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59263
[junit4:junit4]   2> 82769 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 82769 T1192 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59263, initiating session
[junit4:junit4]   2> 82769 T1186 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35654
[junit4:junit4]   2> 82770 T1186 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35654
[junit4:junit4]   2> 82770 T1188 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136197f0001 with negotiated timeout 10000 for client /127.0.0.1:35654
[junit4:junit4]   2> 82770 T1192 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59263, sessionid = 0x13ae136197f0001, negotiated timeout = 10000
[junit4:junit4]   2> 82771 T1193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ac9a93 name:ZooKeeperConnection Watcher:127.0.0.1:59263 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 82771 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 82779 T1189 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136197f0001
[junit4:junit4]   2> 82780 T1186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35654 which had sessionid 0x13ae136197f0001
[junit4:junit4]   2> 82780 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f0001 closed
[junit4:junit4]   2> 82780 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59263 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@161b30c
[junit4:junit4]   2> 82781 T1193 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82781 T1194 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59263
[junit4:junit4]   2> 82781 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 82782 T1194 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59263, initiating session
[junit4:junit4]   2> 82782 T1186 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35655
[junit4:junit4]   2> 82782 T1186 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35655
[junit4:junit4]   2> 82782 T1188 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136197f0002 with negotiated timeout 10000 for client /127.0.0.1:35655
[junit4:junit4]   2> 82782 T1194 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59263, sessionid = 0x13ae136197f0002, negotiated timeout = 10000
[junit4:junit4]   2> 82783 T1195 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@161b30c name:ZooKeeperConnection Watcher:127.0.0.1:59263 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 82783 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 82783 T1143 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 82785 T1189 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136197f0002
[junit4:junit4]   2> 82785 T1195 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82785 T1186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35655 which had sessionid 0x13ae136197f0002
[junit4:junit4]   2> 82785 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f0002 closed
[junit4:junit4]   2> 82786 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 82787 T1143 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 82788 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 82789 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59263/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@34054f
[junit4:junit4]   2> 82790 T1196 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59263
[junit4:junit4]   2> 82790 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 82790 T1196 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59263, initiating session
[junit4:junit4]   2> 82790 T1186 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35656
[junit4:junit4]   2> 82791 T1186 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35656
[junit4:junit4]   2> 82791 T1188 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136197f0003 with negotiated timeout 10000 for client /127.0.0.1:35656
[junit4:junit4]   2> 82791 T1196 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59263, sessionid = 0x13ae136197f0003, negotiated timeout = 10000
[junit4:junit4]   2> 82792 T1197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34054f name:ZooKeeperConnection Watcher:127.0.0.1:59263/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 82792 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 82792 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 82793 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 82795 T1191 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 82795 T1197 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 82795 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59263/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1ac84ed
[junit4:junit4]   2> 82796 T1198 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59263
[junit4:junit4]   2> 82796 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 82796 T1198 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59263, initiating session
[junit4:junit4]   2> 82796 T1186 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35657
[junit4:junit4]   2> 82797 T1186 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35657
[junit4:junit4]   2> 82797 T1188 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136197f0004 with negotiated timeout 10000 for client /127.0.0.1:35657
[junit4:junit4]   2> 82797 T1198 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59263, sessionid = 0x13ae136197f0004, negotiated timeout = 10000
[junit4:junit4]   2> 82797 T1199 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ac84ed name:ZooKeeperConnection Watcher:127.0.0.1:59263/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 82798 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 82798 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 82801 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 82801 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 82802 T1143 oasc.Overseer.start Overseer (id=88630765159383044-127.0.0.1:59263_solr-n_0000000000) starting
[junit4:junit4]   2> 82803 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 82804 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 82804 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 82805 T1201 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 82805 T1200 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 82900 T1181 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 
[junit4:junit4]   2> 82901 T1181 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88630759825276933-127.0.0.1:50631_solr-n_0000000002) am no longer a leader.
[junit4:junit4]   2> 83805 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 83806 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 83808 T1200 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83809 T1200 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 83809 T1200 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 83809 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0004 type:create cxid:0x23 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 83812 T1191 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 83812 T1197 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 84308 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 84312 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 84313 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 84315 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 84316 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84317 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 84318 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 84319 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0x2e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 84815 T1200 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84831 T1200 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 84834 T1191 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 84834 T1197 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 84885 T1189 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136197f0004
[junit4:junit4]   2> 84886 T1199 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 84886 T1186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35657 which had sessionid 0x13ae136197f0004
[junit4:junit4]   2> 84886 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f0004 closed
[junit4:junit4]   2> 85342 T1200 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 
[junit4:junit4]   2> 85342 T1200 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88630765159383044-127.0.0.1:59263_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 85887 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0x34 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85888 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 85890 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0x3b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 85891 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59263/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@fdbc9
[junit4:junit4]   2> 85892 T1202 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59263
[junit4:junit4]   2> 85892 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 85893 T1202 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59263, initiating session
[junit4:junit4]   2> 85893 T1186 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35674
[junit4:junit4]   2> 85893 T1186 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35674
[junit4:junit4]   2> 85894 T1188 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136197f0005 with negotiated timeout 10000 for client /127.0.0.1:35674
[junit4:junit4]   2> 85894 T1202 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59263, sessionid = 0x13ae136197f0005, negotiated timeout = 10000
[junit4:junit4]   2> 85894 T1203 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fdbc9 name:ZooKeeperConnection Watcher:127.0.0.1:59263/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 85894 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 85895 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 85896 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 85897 T1143 oasc.Overseer.start Overseer (id=88630765159383045-127.0.0.1:59263_solr-n_0000000002) starting
[junit4:junit4]   2> 85897 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0005 type:create cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85897 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85898 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0005 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85898 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0005 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85899 T1205 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 85899 T1204 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 85900 T1204 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85901 T1204 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 85902 T1191 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 85902 T1197 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 85954 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0x41 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85955 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 85956 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0x48 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 86404 T1204 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 86408 T1197 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 86409 T1189 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136197f0003 type:create cxid:0x54 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 86409 T1197 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leaders/shard1
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:643)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:399)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:396)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:353)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:340)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:95)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:155)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:99)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 86411 T1197 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 86411 T1197 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 86412 T1191 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 86412 T1197 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 86913 T1191 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 86913 T1189 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136197f0003
[junit4:junit4]   2> 86913 T1197 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 86913 T1197 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 86914 T1186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35656 which had sessionid 0x13ae136197f0003
[junit4:junit4]   2> 86914 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f0003 closed
[junit4:junit4]   2> 86914 T1189 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136197f0005
[junit4:junit4]   2> 86914 T1191 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 86915 T1203 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 86915 T1186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35674 which had sessionid 0x13ae136197f0005
[junit4:junit4]   2> 86915 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f0005 closed
[junit4:junit4]   2> 86915 T1189 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136197f0000
[junit4:junit4]   2> 86916 T1191 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 86916 T1186 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35653 which had sessionid 0x13ae136197f0000
[junit4:junit4]   2> 86916 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f0000 closed
[junit4:junit4]   2> 86916 T1188 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 86916 T1189 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 86916 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 86918 T1186 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 86918 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 86918 T1143 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4:junit4]   2> 86922 T1143 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4:junit4]   2> 86922 T1143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 86923 T1206 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 86923 T1206 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 86924 T1206 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version-2/snapshot.91
[junit4:junit4]   2> 86927 T1206 oazsp.FileTxnSnapLog.save Snapshotting: fd
[junit4:junit4]   2> 87023 T1143 oasc.ZkTestServer.run start zk server on port:35091
[junit4:junit4]   2> 87023 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35091/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1ef3259
[junit4:junit4]   2> 87024 T1211 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:35091
[junit4:junit4]   2> 87024 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87024 T1211 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35091, initiating session
[junit4:junit4]   2> 87025 T1207 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53265
[junit4:junit4]   2> 87025 T1207 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53265
[junit4:junit4]   2> 87025 T1209 oazsp.FileTxnLog.append Creating new log file: log.fe
[junit4:junit4]   2> 87026 T1209 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362a220000 with negotiated timeout 10000 for client /127.0.0.1:53265
[junit4:junit4]   2> 87026 T1211 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:35091, sessionid = 0x13ae1362a220000, negotiated timeout = 10000
[junit4:junit4]   2> 87026 T1212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ef3259 name:ZooKeeperConnection Watcher:127.0.0.1:35091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87027 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87027 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35091 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@55ca25
[junit4:junit4]   2> 87027 T1213 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:35091
[junit4:junit4]   2> 87027 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87028 T1213 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35091, initiating session
[junit4:junit4]   2> 87028 T1207 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53266
[junit4:junit4]   2> 87028 T1207 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53266
[junit4:junit4]   2> 87029 T1209 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362a220001 with negotiated timeout 10000 for client /127.0.0.1:53266
[junit4:junit4]   2> 87029 T1213 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:35091, sessionid = 0x13ae1362a220001, negotiated timeout = 10000
[junit4:junit4]   2> 87029 T1214 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55ca25 name:ZooKeeperConnection Watcher:127.0.0.1:35091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87029 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87034 T1210 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362a220001
[junit4:junit4]   2> 87035 T1214 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 87035 T1207 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53266 which had sessionid 0x13ae1362a220001
[junit4:junit4]   2> 87035 T1143 oaz.ZooKeeper.close Session: 0x13ae1362a220001 closed
[junit4:junit4]   2> 87035 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35091 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@862f9b
[junit4:junit4]   2> 87036 T1215 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:35091
[junit4:junit4]   2> 87036 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87036 T1215 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35091, initiating session
[junit4:junit4]   2> 87036 T1207 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53267
[junit4:junit4]   2> 87036 T1207 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53267
[junit4:junit4]   2> 87037 T1209 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362a220002 with negotiated timeout 10000 for client /127.0.0.1:53267
[junit4:junit4]   2> 87037 T1215 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:35091, sessionid = 0x13ae1362a220002, negotiated timeout = 10000
[junit4:junit4]   2> 87037 T1216 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@862f9b name:ZooKeeperConnection Watcher:127.0.0.1:35091 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87037 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87038 T1143 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 87040 T1210 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362a220002
[junit4:junit4]   2> 87041 T1216 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 87041 T1207 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53267 which had sessionid 0x13ae1362a220002
[junit4:junit4]   2> 87041 T1143 oaz.ZooKeeper.close Session: 0x13ae1362a220002 closed
[junit4:junit4]   2> 87041 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 87043 T1143 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 87044 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 87044 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35091/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1498f23
[junit4:junit4]   2> 87045 T1217 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:35091
[junit4:junit4]   2> 87045 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87045 T1217 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:35091, initiating session
[junit4:junit4]   2> 87045 T1207 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53268
[junit4:junit4]   2> 87046 T1207 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53268
[junit4:junit4]   2> 87046 T1209 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362a220003 with negotiated timeout 10000 for client /127.0.0.1:53268
[junit4:junit4]   2> 87046 T1217 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:35091, sessionid = 0x13ae1362a220003, negotiated timeout = 10000
[junit4:junit4]   2> 87046 T1218 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1498f23 name:ZooKeeperConnection Watcher:127.0.0.1:35091/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87047 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87047 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 87049 T1210 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362a220003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 87050 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 87051 T1143 oasc.Overseer.start Overseer (id=88630765438500867-127.0.0.1:35091_solr-n_0000000000) starting
[junit4:junit4]   2> 87051 T1210 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362a220003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87052 T1210 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362a220003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87052 T1210 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362a220003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87053 T1220 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 87053 T1210 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362a220000 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87054 T1210 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362a220000 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 87054 T1219 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 87055 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87055 T1219 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87055 T1219 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr"}
[junit4:junit4]   2> 87056 T1210 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362a220003 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 87058 T1212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 87106 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87414 T1204 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 
[junit4:junit4]   2> 87415 T1204 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88630765159383045-127.0.0.1:59263_solr-n_0000000002) am no longer a leader.
[junit4:junit4]   2> 87560 T1219 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87561 T1219 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr"}
[junit4:junit4]   2> 87563 T1212 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 87608 T1210 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362a220000
[junit4:junit4]   2> 87609 T1212 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 87609 T1207 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53265 which had sessionid 0x13ae1362a220000
[junit4:junit4]   2> 87609 T1143 oaz.ZooKeeper.close Session: 0x13ae1362a220000 closed
[junit4:junit4]   2> 87610 T1210 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362a220003
[junit4:junit4]   2> 87610 T1218 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 87610 T1207 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53268 which had sessionid 0x13ae1362a220003
[junit4:junit4]   2> 87610 T1143 oaz.ZooKeeper.close Session: 0x13ae1362a220003 closed
[junit4:junit4]   2> 87611 T1210 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 87611 T1209 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 87611 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 87612 T1207 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 87613 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 87613 T1143 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4:junit4]   2> 87617 T1143 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
[junit4:junit4]   2> 87617 T1143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 87617 T1221 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 87618 T1221 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 87618 T1221 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version-2/snapshot.fd
[junit4:junit4]   2> 87621 T1221 oazsp.FileTxnSnapLog.save Snapshotting: 12b
[junit4:junit4]   2> 87654 T1187 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 87654 T1208 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 87717 T1143 oasc.ZkTestServer.run start zk server on port:57640
[junit4:junit4]   2> 87718 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:57640/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@b0ecde
[junit4:junit4]   2> 87719 T1226 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:57640
[junit4:junit4]   2> 87719 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87719 T1226 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:57640, initiating session
[junit4:junit4]   2> 87719 T1222 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:38742
[junit4:junit4]   2> 87719 T1222 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:38742
[junit4:junit4]   2> 87720 T1224 oazsp.FileTxnLog.append Creating new log file: log.12c
[junit4:junit4]   2> 87722 T1224 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362cd70000 with negotiated timeout 10000 for client /127.0.0.1:38742
[junit4:junit4]   2> 87722 T1226 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:57640, sessionid = 0x13ae1362cd70000, negotiated timeout = 10000
[junit4:junit4]   2> 87723 T1227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b0ecde name:ZooKeeperConnection Watcher:127.0.0.1:57640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87723 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87723 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:57640 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1152a31
[junit4:junit4]   2> 87724 T1228 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:57640
[junit4:junit4]   2> 87724 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87724 T1228 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:57640, initiating session
[junit4:junit4]   2> 87724 T1222 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:38743
[junit4:junit4]   2> 87725 T1222 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:38743
[junit4:junit4]   2> 87725 T1224 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362cd70001 with negotiated timeout 10000 for client /127.0.0.1:38743
[junit4:junit4]   2> 87725 T1228 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:57640, sessionid = 0x13ae1362cd70001, negotiated timeout = 10000
[junit4:junit4]   2> 87726 T1229 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1152a31 name:ZooKeeperConnection Watcher:127.0.0.1:57640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87726 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87731 T1225 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362cd70001
[junit4:junit4]   2> 87732 T1229 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 87732 T1222 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:38743 which had sessionid 0x13ae1362cd70001
[junit4:junit4]   2> 87732 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd70001 closed
[junit4:junit4]   2> 87732 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:57640 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1ed5d72
[junit4:junit4]   2> 87733 T1230 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:57640
[junit4:junit4]   2> 87733 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87733 T1230 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:57640, initiating session
[junit4:junit4]   2> 87733 T1222 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:38744
[junit4:junit4]   2> 87733 T1222 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:38744
[junit4:junit4]   2> 87734 T1224 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362cd70002 with negotiated timeout 10000 for client /127.0.0.1:38744
[junit4:junit4]   2> 87734 T1230 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:57640, sessionid = 0x13ae1362cd70002, negotiated timeout = 10000
[junit4:junit4]   2> 87734 T1231 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ed5d72 name:ZooKeeperConnection Watcher:127.0.0.1:57640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87734 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87735 T1143 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 87736 T1225 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362cd70002
[junit4:junit4]   2> 87736 T1231 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 87736 T1222 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:38744 which had sessionid 0x13ae1362cd70002
[junit4:junit4]   2> 87736 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd70002 closed
[junit4:junit4]   2> 87737 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 87738 T1143 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 87739 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 87740 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:57640/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@17491f8
[junit4:junit4]   2> 87741 T1232 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:57640
[junit4:junit4]   2> 87741 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87741 T1232 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:57640, initiating session
[junit4:junit4]   2> 87741 T1222 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:38745
[junit4:junit4]   2> 87741 T1222 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:38745
[junit4:junit4]   2> 87742 T1224 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362cd70003 with negotiated timeout 10000 for client /127.0.0.1:38745
[junit4:junit4]   2> 87742 T1232 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:57640, sessionid = 0x13ae1362cd70003, negotiated timeout = 10000
[junit4:junit4]   2> 87742 T1233 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17491f8 name:ZooKeeperConnection Watcher:127.0.0.1:57640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87742 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87750 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 87750 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 87753 T1227 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 87753 T1233 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 87753 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:57640/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1527903
[junit4:junit4]   2> 87754 T1234 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:57640
[junit4:junit4]   2> 87754 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87754 T1234 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:57640, initiating session
[junit4:junit4]   2> 87754 T1222 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:38746
[junit4:junit4]   2> 87754 T1222 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:38746
[junit4:junit4]   2> 87755 T1224 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362cd70004 with negotiated timeout 10000 for client /127.0.0.1:38746
[junit4:junit4]   2> 87755 T1234 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:57640, sessionid = 0x13ae1362cd70004, negotiated timeout = 10000
[junit4:junit4]   2> 87755 T1235 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1527903 name:ZooKeeperConnection Watcher:127.0.0.1:57640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87755 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87756 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 87758 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 87759 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 87760 T1143 oasc.Overseer.start Overseer (id=88630765483917316-127.0.0.1:57640_solr-n_0000000000) starting
[junit4:junit4]   2> 87760 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87761 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87761 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87762 T1237 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 87762 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87763 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 87763 T1236 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 87914 T1197 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 87914 T1197 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 88064 T1219 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 
[junit4:junit4]   2> 88065 T1219 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88630765438500867-127.0.0.1:35091_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 88265 T1236 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88266 T1236 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 88266 T1236 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 88266 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70004 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 88269 T1227 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 88269 T1233 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 88764 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 88768 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70003 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 88769 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 88771 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70003 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 88772 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88773 T1233 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 88773 T1225 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362cd70003
[junit4:junit4]   2> 88773 T1227 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 88773 T1227 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 88773 T1222 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:38745 which had sessionid 0x13ae1362cd70003
[junit4:junit4]   2> 88773 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd70003 closed
[junit4:junit4]   2> 88774 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:57640/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1d70b1d
[junit4:junit4]   2> 88775 T1238 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:57640
[junit4:junit4]   2> 88775 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 88775 T1238 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:57640, initiating session
[junit4:junit4]   2> 88775 T1222 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:38747
[junit4:junit4]   2> 88776 T1222 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:38747
[junit4:junit4]   2> 88776 T1238 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:57640, sessionid = 0x13ae1362cd70005, negotiated timeout = 10000
[junit4:junit4]   2> 88776 T1224 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae1362cd70005 with negotiated timeout 10000 for client /127.0.0.1:38747
[junit4:junit4]   2> 88777 T1239 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d70b1d name:ZooKeeperConnection Watcher:127.0.0.1:57640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 88777 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 88777 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 88778 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 88780 T1227 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 88780 T1239 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 88780 T1227 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 88780 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 88781 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 88782 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70005 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 88783 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70005 type:delete cxid:0x14 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard1 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard1
[junit4:junit4]   2> 88783 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 88785 T1225 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae1362cd70005 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89272 T1236 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89274 T1236 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 89279 T1227 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 89279 T1239 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 89280 T1143 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89282 T1225 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362cd70004
[junit4:junit4]   2> 89283 T1235 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89282 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd70004 closed
[junit4:junit4]   2> 89283 T1222 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:38746 which had sessionid 0x13ae1362cd70004
[junit4:junit4]   2> 89284 T1239 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 89285 T1225 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362cd70005
[junit4:junit4]   2> 89285 T1222 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:38747 which had sessionid 0x13ae1362cd70005
[junit4:junit4]   2> 89285 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd70005 closed
[junit4:junit4]   2> 89286 T1227 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 89286 T1225 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae1362cd70000
[junit4:junit4]   2> 89286 T1227 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 89286 T1222 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:38742 which had sessionid 0x13ae1362cd70000
[junit4:junit4]   2> 89286 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd70000 closed
[junit4:junit4]   2> 89287 T1225 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 89287 T1224 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 89287 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 89289 T1222 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 89289 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 89290 T1143 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
[junit4:junit4]   2> 89296 T1143 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4:junit4]   2> 89296 T1143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 89297 T1240 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 89297 T1240 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 89300 T1240 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version-2/snapshot.12b
[junit4:junit4]   2> 89303 T1240 oazsp.FileTxnSnapLog.save Snapshotting: 176
[junit4:junit4]   2> 89397 T1143 oasc.ZkTestServer.run start zk server on port:42634
[junit4:junit4]   2> 89398 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42634/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1644431
[junit4:junit4]   2> 89399 T1245 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42634
[junit4:junit4]   2> 89399 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89399 T1241 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:48715
[junit4:junit4]   2> 89399 T1245 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42634, initiating session
[junit4:junit4]   2> 89400 T1241 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:48715
[junit4:junit4]   2> 89400 T1243 oazsp.FileTxnLog.append Creating new log file: log.177
[junit4:junit4]   2> 89413 T1245 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42634, sessionid = 0x13ae136336a0000, negotiated timeout = 10000
[junit4:junit4]   2> 89413 T1243 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136336a0000 with negotiated timeout 10000 for client /127.0.0.1:48715
[junit4:junit4]   2> 89413 T1246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1644431 name:ZooKeeperConnection Watcher:127.0.0.1:42634/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89413 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89414 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42634 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@12b0019
[junit4:junit4]   2> 89415 T1247 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42634
[junit4:junit4]   2> 89415 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89415 T1247 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42634, initiating session
[junit4:junit4]   2> 89415 T1241 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:48716
[junit4:junit4]   2> 89415 T1241 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:48716
[junit4:junit4]   2> 89426 T1247 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42634, sessionid = 0x13ae136336a0001, negotiated timeout = 10000
[junit4:junit4]   2> 89426 T1243 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136336a0001 with negotiated timeout 10000 for client /127.0.0.1:48716
[junit4:junit4]   2> 89426 T1248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12b0019 name:ZooKeeperConnection Watcher:127.0.0.1:42634 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89426 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89434 T1244 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136336a0001
[junit4:junit4]   2> 89435 T1248 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89435 T1241 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:48716 which had sessionid 0x13ae136336a0001
[junit4:junit4]   2> 89435 T1143 oaz.ZooKeeper.close Session: 0x13ae136336a0001 closed
[junit4:junit4]   2> 89450 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42634 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@162e705
[junit4:junit4]   2> 89451 T1249 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42634
[junit4:junit4]   2> 89451 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89452 T1249 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42634, initiating session
[junit4:junit4]   2> 89452 T1241 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:48717
[junit4:junit4]   2> 89452 T1241 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:48717
[junit4:junit4]   2> 89453 T1249 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42634, sessionid = 0x13ae136336a0002, negotiated timeout = 10000
[junit4:junit4]   2> 89453 T1243 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136336a0002 with negotiated timeout 10000 for client /127.0.0.1:48717
[junit4:junit4]   2> 89454 T1250 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@162e705 name:ZooKeeperConnection Watcher:127.0.0.1:42634 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89454 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89454 T1143 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 89456 T1244 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136336a0002
[junit4:junit4]   2> 89456 T1250 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89456 T1241 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:48717 which had sessionid 0x13ae136336a0002
[junit4:junit4]   2> 89456 T1143 oaz.ZooKeeper.close Session: 0x13ae136336a0002 closed
[junit4:junit4]   2> 89457 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 89458 T1143 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 89459 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 89460 T1244 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136336a0000 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 89462 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42634/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@e5b7ca
[junit4:junit4]   2> 89462 T1251 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42634
[junit4:junit4]   2> 89462 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89463 T1251 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42634, initiating session
[junit4:junit4]   2> 89463 T1241 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:48718
[junit4:junit4]   2> 89463 T1241 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:48718
[junit4:junit4]   2> 89464 T1243 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae136336a0003 with negotiated timeout 10000 for client /127.0.0.1:48718
[junit4:junit4]   2> 89464 T1251 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42634, sessionid = 0x13ae136336a0003, negotiated timeout = 10000
[junit4:junit4]   2> 89464 T1252 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e5b7ca name:ZooKeeperConnection Watcher:127.0.0.1:42634/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89464 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89465 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 89466 T1244 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136336a0003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 89467 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 89468 T1143 oasc.Overseer.start Overseer (id=88630765594214403-127.0.0.1:42634_solr-n_0000000000) starting
[junit4:junit4]   2> 89468 T1244 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136336a0003 type:create cxid:0xe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89469 T1244 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136336a0003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89469 T1244 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136336a0003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89469 T1244 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136336a0003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89471 T1254 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 89471 T1244 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136336a0000 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89471 T1244 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae136336a0000 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 89473 T1253 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89473 T1253 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4:junit4]   2> 89473 T1253 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":"s1",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr"}
[junit4:junit4]   2> 89474 T1246 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 89475 T1253 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":"s1",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"core2",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr"}
[junit4:junit4]   2> 89476 T1246 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 89477 T1253 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 89477 T1253 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89477 T1253 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "shard":"s1",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"core3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr"}
[junit4:junit4]   2> 89479 T1246 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 89573 T1244 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136336a0003
[junit4:junit4]   2> 89574 T1252 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89574 T1241 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:48718 which had sessionid 0x13ae136336a0003
[junit4:junit4]   2> 89574 T1143 oaz.ZooKeeper.close Session: 0x13ae136336a0003 closed
[junit4:junit4]   2> 89574 T1244 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae136336a0000
[junit4:junit4]   2> 89575 T1246 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89575 T1241 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:48715 which had sessionid 0x13ae136336a0000
[junit4:junit4]   2> 89575 T1143 oaz.ZooKeeper.close Session: 0x13ae136336a0000 closed
[junit4:junit4]   2> 89575 T1244 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 89575 T1243 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 89576 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 89577 T1241 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 89577 T1143 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 89578 T1143 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4:junit4]   2> 89582 T1143 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4:junit4]   2> 89582 T1143 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 89583 T1255 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 89583 T1255 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 89583 T1255 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version-2/snapshot.176
[junit4:junit4]   2> 89588 T1255 oazsp.FileTxnSnapLog.save Snapshotting: 1ad
[junit4:junit4]   2> 89654 T1223 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 89654 T1242 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 89683 T1143 oasc.ZkTestServer.run start zk server on port:51735
[junit4:junit4]   2> 89683 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51735 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@10307f7
[junit4:junit4]   2> 89684 T1260 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:51735
[junit4:junit4]   2> 89684 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89684 T1260 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51735, initiating session
[junit4:junit4]   2> 89684 T1256 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:49784
[junit4:junit4]   2> 89685 T1256 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:49784
[junit4:junit4]   2> 89685 T1258 oazsp.FileTxnLog.append Creating new log file: log.1ae
[junit4:junit4]   2> 89686 T1258 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae13634860000 with negotiated timeout 10000 for client /127.0.0.1:49784
[junit4:junit4]   2> 89686 T1260 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:51735, sessionid = 0x13ae13634860000, negotiated timeout = 10000
[junit4:junit4]   2> 89686 T1261 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10307f7 name:ZooKeeperConnection Watcher:127.0.0.1:51735 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89687 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89691 T1259 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae13634860000
[junit4:junit4]   2> 89692 T1256 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:49784 which had sessionid 0x13ae13634860000
[junit4:junit4]   2> 89692 T1143 oaz.ZooKeeper.close Session: 0x13ae13634860000 closed
[junit4:junit4]   2> 89692 T1261 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89692 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51735 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1d66a30
[junit4:junit4]   2> 89693 T1262 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:51735
[junit4:junit4]   2> 89693 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89693 T1256 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:49785
[junit4:junit4]   2> 89693 T1262 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51735, initiating session
[junit4:junit4]   2> 89693 T1256 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:49785
[junit4:junit4]   2> 89694 T1258 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae13634860001 with negotiated timeout 10000 for client /127.0.0.1:49785
[junit4:junit4]   2> 89694 T1262 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:51735, sessionid = 0x13ae13634860001, negotiated timeout = 10000
[junit4:junit4]   2> 89694 T1263 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d66a30 name:ZooKeeperConnection Watcher:127.0.0.1:51735 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89694 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89695 T1143 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 89696 T1259 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ae13634860001
[junit4:junit4]   2> 89696 T1256 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:49785 which had sessionid 0x13ae13634860001
[junit4:junit4]   2> 89696 T1143 oaz.ZooKeeper.close Session: 0x13ae13634860001 closed
[junit4:junit4]   2> 89697 T1263 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89697 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51735/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@7a8dfc
[junit4:junit4]   2> 89697 T1264 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:51735
[junit4:junit4]   2> 89697 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89698 T1256 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:49786
[junit4:junit4]   2> 89698 T1264 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51735, initiating session
[junit4:junit4]   2> 89698 T1256 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:49786
[junit4:junit4]   2> 89700 T1258 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae13634860002 with negotiated timeout 10000 for client /127.0.0.1:49786
[junit4:junit4]   2> 89700 T1264 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:51735, sessionid = 0x13ae13634860002, negotiated timeout = 10000
[junit4:junit4]   2> 89700 T1265 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a8dfc name:ZooKeeperConnection Watcher:127.0.0.1:51735/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89700 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89700 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 89702 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51735/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@5bbeb7
[junit4:junit4]   2> 89702 T1266 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:51735
[junit4:junit4]   2> 89702 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89703 T1266 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51735, initiating session
[junit4:junit4]   2> 89703 T1256 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:49787
[junit4:junit4]   2> 89703 T1256 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:49787
[junit4:junit4]   2> 89704 T1258 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae13634860003 with negotiated timeout 10000 for client /127.0.0.1:49787
[junit4:junit4]   2> 89704 T1266 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:51735, sessionid = 0x13ae13634860003, negotiated timeout = 10000
[junit4:junit4]   2> 89704 T1267 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bbeb7 name:ZooKeeperConnection Watcher:127.0.0.1:51735/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89704 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89705 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 89707 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 89707 T1143 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 89708 T1143 oasc.Overseer.start Overseer (id=88630765612826627-127.0.0.1:51735_solr-n_0000000000) starting
[junit4:junit4]   2> 89709 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89709 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89710 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89711 T1269 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 89711 T1143 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 89712 T1268 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 89713 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 89714 T1143 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:51735/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@9161f0
[junit4:junit4]   2> 89714 T1270 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:51735
[junit4:junit4]   2> 89714 T1143 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89715 T1270 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:51735, initiating session
[junit4:junit4]   2> 89715 T1256 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:49788
[junit4:junit4]   2> 89715 T1256 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:49788
[junit4:junit4]   2> 89716 T1258 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ae13634860004 with negotiated timeout 10000 for client /127.0.0.1:49788
[junit4:junit4]   2> 89716 T1270 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:51735, sessionid = 0x13ae13634860004, negotiated timeout = 10000
[junit4:junit4]   2> 89716 T1271 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9161f0 name:ZooKeeperConnection Watcher:127.0.0.1:51735/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89716 T1143 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89717 T1143 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 89718 T1143 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4:junit4]   2> 89721 T1271 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 89721 T1265 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 89721 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860004 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89721 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860004 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 89773 T1233 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 89773 T1233 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 89774 T1233 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89980 T1253 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2> 
[junit4:junit4]   2> 89980 T1253 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88630765594214403-127.0.0.1:42634_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 90214 T1268 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90214 T1268 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
[junit4:junit4]   2> 90215 T1268 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4]   2> 90215 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860003 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 90218 T1271 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 90218 T1265 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 90222 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 90227 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860004 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 90227 T1143 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 90229 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860004 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 90230 T1259 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ae13634860004 type:create cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 90282 T1236 oasc.Overseer$ClusterStateUpdater.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 90285 T1239 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 90285 T1239 oasc

[...truncated too long message...]

Keeper.java:815)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:176)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:173)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:173)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:384)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:353)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:340)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:95)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:155)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:99)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 103215 T1414 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103216 T1414 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (18)
[junit4:junit4]   2> 103216 T1414 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103216 T1414 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103223 T1416 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103223 T1416 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103228 T1418 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103228 T1418 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103229 T1418 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103252 T1422 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103253 T1422 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103271 T1424 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103272 T1424 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103293 T1426 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:643)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:210)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:207)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:207)
[junit4:junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:226)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:104)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:155)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:99)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 103293 T1426 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (13)
[junit4:junit4]   2> 103293 T1426 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103293 T1426 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103293 T1426 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103300 T1430 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103300 T1430 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103301 T1430 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103304 T1432 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103305 T1432 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103305 T1432 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103315 T1436 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103315 T1436 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103316 T1436 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103332 T1438 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103333 T1438 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103335 T1440 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103335 T1440 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103340 T1442 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103341 T1442 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103343 T1444 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103344 T1444 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 103345 T1446 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103345 T1446 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 103345 T1446 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 105350 T1143 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=nl_NL, timezone=America/Aruba
[junit4:junit4]   2> NOTE: Linux 3.2.0-32-generic i386/Sun Microsystems Inc. 1.6.0_37 (32-bit)/cpus=8,threads=10,free=210099408,total=302907392
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestSort, TestCSVResponseWriter, TestFunctionQuery, ClusterStateTest, ResponseLogComponentTest, LegacyHTMLStripCharFilterTest, FieldMutatingUpdateProcessorTest, TestSolrDeletionPolicy1, OutputWriterTest, MultiTermTest, SyncSliceTest, TestDistributedSearch, ShowFileRequestHandlerTest, TestQuerySenderListener, RecoveryZkTest, IndexSchemaRuntimeFieldTest, XsltUpdateRequestHandlerTest, TestPhraseSuggestions, PeerSyncTest, SolrInfoMBeanTest, TestDistributedGrouping, TestPluginEnable, DistributedQueryElevationComponentTest, FullSolrCloudDistribCmdsTest, TestReversedWildcardFilterFactory, DebugComponentTest, TestFoldingMultitermQuery, TestRecovery, TestWriterPerf, TestQueryUtils, ConvertedLegacyTest, TestBinaryField, XmlUpdateRequestHandlerTest, TestSolrDeletionPolicy2, TestSurroundQueryParser, DocumentBuilderTest, DirectUpdateHandlerTest, DocumentAnalysisRequestHandlerTest, StatsComponentTest, TestStressVersions, FileUtilsTest, BasicZkTest, TestValueSourceCache, TimeZoneUtilsTest, PrimUtilsTest, TestDocSet, TestSolrJ, SuggesterTest, QueryEqualityTest, ZkNodePropsTest, SignatureUpdateProcessorFactoryTest, DirectSolrSpellCheckerTest, OverseerTest]
[junit4:junit4] Completed on J1 in 105.37s, 8 tests, 1 failure <<< FAILURES!

[...truncated 375 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:335: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:178: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1163: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:827: There were test failures: 233 suites, 970 tests, 1 failure, 7 ignored (2 assumptions)

Total time: 24 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/jdk1.6.0_37 -server -XX:+UseParallelGC
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message