lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@sd-datasolutions.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b58) - Build # 2155 - Failure!
Date Fri, 02 Nov 2012 15:46:24 GMT
Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Linux/2155/
Java: 32bit/jdk1.8.0-ea-b58 -client -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testShardLeaderChange

Error Message:
Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>

Stack Trace:
org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>
	at __randomizedtesting.SeedInfo.seed([B8C7B18B86CE8EE3:6694367C9C567B12]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:493)
	at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:652)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:474)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 8761 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4:junit4]   2> 0 T1178 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351870787193
[junit4:junit4]   2> 1 T1178 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 3 T1178 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
[junit4:junit4]   2> 4 T1178 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 5 T1179 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 5 T1179 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 6 T1179 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 104 T1178 oasc.ZkTestServer.run start zk server on port:59194
[junit4:junit4]   2> 105 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59194/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@18ab7c3
[junit4:junit4]   2> 106 T1184 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59194
[junit4:junit4]   2> 106 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 106 T1184 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59194, initiating session
[junit4:junit4]   2> 107 T1180 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58942
[junit4:junit4]   2> 107 T1180 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58942
[junit4:junit4]   2> 107 T1182 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 190 T1182 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c72e7e0000 with negotiated timeout 10000 for client /127.0.0.1:58942
[junit4:junit4]   2> 190 T1184 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59194, sessionid = 0x13ac1c72e7e0000, negotiated timeout = 10000
[junit4:junit4]   2> 191 T1185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18ab7c3 name:ZooKeeperConnection Watcher:127.0.0.1:59194/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 191 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 191 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59194 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1fea4d5
[junit4:junit4]   2> 192 T1186 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59194
[junit4:junit4]   2> 192 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 193 T1186 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59194, initiating session
[junit4:junit4]   2> 193 T1180 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58943
[junit4:junit4]   2> 193 T1180 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58943
[junit4:junit4]   2> 195 T1182 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c72e7e0001 with negotiated timeout 10000 for client /127.0.0.1:58943
[junit4:junit4]   2> 195 T1186 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59194, sessionid = 0x13ac1c72e7e0001, negotiated timeout = 10000
[junit4:junit4]   2> 195 T1187 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fea4d5 name:ZooKeeperConnection Watcher:127.0.0.1:59194 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 196 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 197 T1183 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c72e7e0001
[junit4:junit4]   2> 197 T1187 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 197 T1180 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58943 which had sessionid 0x13ac1c72e7e0001
[junit4:junit4]   2> 197 T1178 oaz.ZooKeeper.close Session: 0x13ac1c72e7e0001 closed
[junit4:junit4]   2> 198 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59194 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@143c4c8
[junit4:junit4]   2> 199 T1188 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59194
[junit4:junit4]   2> 199 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 199 T1188 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59194, initiating session
[junit4:junit4]   2> 200 T1180 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58944
[junit4:junit4]   2> 200 T1180 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58944
[junit4:junit4]   2> 201 T1182 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c72e7e0002 with negotiated timeout 10000 for client /127.0.0.1:58944
[junit4:junit4]   2> 201 T1188 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59194, sessionid = 0x13ac1c72e7e0002, negotiated timeout = 10000
[junit4:junit4]   2> 201 T1189 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@143c4c8 name:ZooKeeperConnection Watcher:127.0.0.1:59194 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 201 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 202 T1178 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 204 T1183 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c72e7e0002
[junit4:junit4]   2> 205 T1180 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58944 which had sessionid 0x13ac1c72e7e0002
[junit4:junit4]   2> 205 T1189 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 205 T1178 oaz.ZooKeeper.close Session: 0x13ac1c72e7e0002 closed
[junit4:junit4]   2> 205 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 207 T1178 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 208 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 210 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59194/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1ee833e
[junit4:junit4]   2> 211 T1190 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59194
[junit4:junit4]   2> 211 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 211 T1190 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59194, initiating session
[junit4:junit4]   2> 211 T1180 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58945
[junit4:junit4]   2> 211 T1180 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58945
[junit4:junit4]   2> 212 T1182 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c72e7e0003 with negotiated timeout 10000 for client /127.0.0.1:58945
[junit4:junit4]   2> 212 T1190 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59194, sessionid = 0x13ac1c72e7e0003, negotiated timeout = 10000
[junit4:junit4]   2> 213 T1191 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ee833e name:ZooKeeperConnection Watcher:127.0.0.1:59194/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 213 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 213 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 215 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 216 T1191 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 217 T1185 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 217 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:59194/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@18ae772
[junit4:junit4]   2> 217 T1192 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:59194
[junit4:junit4]   2> 217 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 218 T1192 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:59194, initiating session
[junit4:junit4]   2> 218 T1180 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:58946
[junit4:junit4]   2> 218 T1180 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:58946
[junit4:junit4]   2> 219 T1182 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c72e7e0004 with negotiated timeout 10000 for client /127.0.0.1:58946
[junit4:junit4]   2> 219 T1192 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:59194, sessionid = 0x13ac1c72e7e0004, negotiated timeout = 10000
[junit4:junit4]   2> 219 T1193 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18ae772 name:ZooKeeperConnection Watcher:127.0.0.1:59194/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 219 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 220 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 223 T1183 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c72e7e0004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 223 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 225 T1178 oasc.Overseer.start Overseer (id=88596203909808132-127.0.0.1:59194_solr-n_0000000000) starting
[junit4:junit4]   2> 226 T1183 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c72e7e0004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 226 T1183 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c72e7e0004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 227 T1183 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c72e7e0004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 227 T1195 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 228 T1183 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c72e7e0003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 228 T1194 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 229 T1183 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c72e7e0003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 731 T1194 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 732 T1194 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "numShards":"12",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 732 T1194 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 12
[junit4:junit4]   2> 733 T1183 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c72e7e0004 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 736 T1185 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 736 T1191 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 1230 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 1238 T1183 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c72e7e0003 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 1239 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 1242 T1183 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c72e7e0003 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1243 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1245 T1183 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c72e7e0004
[junit4:junit4]   2> 1245 T1193 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1246 T1180 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58946 which had sessionid 0x13ac1c72e7e0004
[junit4:junit4]   2> 1245 T1178 oaz.ZooKeeper.close Session: 0x13ac1c72e7e0004 closed
[junit4:junit4]   2> 1247 T1185 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 1247 T1183 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c72e7e0003
[junit4:junit4]   2> 1248 T1180 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58945 which had sessionid 0x13ac1c72e7e0003
[junit4:junit4]   2> 1248 T1178 oaz.ZooKeeper.close Session: 0x13ac1c72e7e0003 closed
[junit4:junit4]   2> 1248 T1183 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c72e7e0000
[junit4:junit4]   2> 1249 T1180 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:58942 which had sessionid 0x13ac1c72e7e0000
[junit4:junit4]   2> 1249 T1178 oaz.ZooKeeper.close Session: 0x13ac1c72e7e0000 closed
[junit4:junit4]   2> 1250 T1182 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 1250 T1183 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 1250 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 1252 T1180 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 1253 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 1253 T1178 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
[junit4:junit4]   2> 1257 T1178 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
[junit4:junit4]   2> 1258 T1178 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 1259 T1196 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 1259 T1196 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 1259 T1196 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351870787193/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 1261 T1196 oazsp.FileTxnSnapLog.save Snapshotting: 2e
[junit4:junit4]   2> 1359 T1178 oasc.ZkTestServer.run start zk server on port:38243
[junit4:junit4]   2> 1359 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:38243/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@12a4d51
[junit4:junit4]   2> 1360 T1201 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:38243
[junit4:junit4]   2> 1360 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1361 T1201 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:38243, initiating session
[junit4:junit4]   2> 1361 T1197 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60153
[junit4:junit4]   2> 1361 T1197 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60153
[junit4:junit4]   2> 1361 T1199 oazsp.FileTxnLog.append Creating new log file: log.2f
[junit4:junit4]   2> 1470 T1201 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:38243, sessionid = 0x13ac1c733650000, negotiated timeout = 10000
[junit4:junit4]   2> 1470 T1199 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c733650000 with negotiated timeout 10000 for client /127.0.0.1:60153
[junit4:junit4]   2> 1470 T1202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12a4d51 name:ZooKeeperConnection Watcher:127.0.0.1:38243/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1471 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1471 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:38243 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1131b84
[junit4:junit4]   2> 1473 T1203 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:38243
[junit4:junit4]   2> 1473 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1473 T1203 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:38243, initiating session
[junit4:junit4]   2> 1473 T1197 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60154
[junit4:junit4]   2> 1474 T1197 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60154
[junit4:junit4]   2> 1475 T1199 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c733650001 with negotiated timeout 10000 for client /127.0.0.1:60154
[junit4:junit4]   2> 1475 T1203 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:38243, sessionid = 0x13ac1c733650001, negotiated timeout = 10000
[junit4:junit4]   2> 1476 T1204 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1131b84 name:ZooKeeperConnection Watcher:127.0.0.1:38243 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1476 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1488 T1200 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c733650001
[junit4:junit4]   2> 1488 T1204 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1489 T1197 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60154 which had sessionid 0x13ac1c733650001
[junit4:junit4]   2> 1488 T1178 oaz.ZooKeeper.close Session: 0x13ac1c733650001 closed
[junit4:junit4]   2> 1489 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:38243 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1230cfe
[junit4:junit4]   2> 1490 T1205 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:38243
[junit4:junit4]   2> 1490 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1490 T1205 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:38243, initiating session
[junit4:junit4]   2> 1491 T1197 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60155
[junit4:junit4]   2> 1491 T1197 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60155
[junit4:junit4]   2> 1492 T1205 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:38243, sessionid = 0x13ac1c733650002, negotiated timeout = 10000
[junit4:junit4]   2> 1492 T1199 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c733650002 with negotiated timeout 10000 for client /127.0.0.1:60155
[junit4:junit4]   2> 1492 T1206 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1230cfe name:ZooKeeperConnection Watcher:127.0.0.1:38243 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1492 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1493 T1178 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 1495 T1200 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c733650002
[junit4:junit4]   2> 1495 T1206 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1495 T1197 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60155 which had sessionid 0x13ac1c733650002
[junit4:junit4]   2> 1495 T1178 oaz.ZooKeeper.close Session: 0x13ac1c733650002 closed
[junit4:junit4]   2> 1496 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1498 T1207 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:38243/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@28ba44
[junit4:junit4]   2> 1498 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:38243/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1a9e99e
[junit4:junit4]   2> 1498 T1208 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:38243
[junit4:junit4]   2> 1498 T1210 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:38243
[junit4:junit4]   2> 1498 T1207 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1499 T1208 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:38243, initiating session
[junit4:junit4]   2> 1499 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1499 T1210 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:38243, initiating session
[junit4:junit4]   2> 1499 T1197 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60156
[junit4:junit4]   2> 1500 T1197 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60157
[junit4:junit4]   2> 1500 T1197 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60156
[junit4:junit4]   2> 1501 T1197 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60157
[junit4:junit4]   2> 1501 T1199 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c733650003 with negotiated timeout 10000 for client /127.0.0.1:60156
[junit4:junit4]   2> 1501 T1208 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:38243, sessionid = 0x13ac1c733650003, negotiated timeout = 10000
[junit4:junit4]   2> 1501 T1209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28ba44 name:ZooKeeperConnection Watcher:127.0.0.1:38243/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1502 T1210 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:38243, sessionid = 0x13ac1c733650004, negotiated timeout = 10000
[junit4:junit4]   2> 1502 T1199 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c733650004 with negotiated timeout 10000 for client /127.0.0.1:60157
[junit4:junit4]   2> 1502 T1207 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1502 T1211 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a9e99e name:ZooKeeperConnection Watcher:127.0.0.1:38243/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1503 T1207 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1502 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1504 T1178 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1505 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1505 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1506 T1207 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1507 T1207 oasc.Overseer.start Overseer (id=88596203992055811-127.0.0.1:38243_solr-n_0000000000) starting
[junit4:junit4]   2> 1507 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 1508 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1508 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1509 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1510 T1211 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 1510 T1213 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1510 T1207 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4:junit4]   2> 1510 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650004 type:create cxid:0xd zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1511 T1200 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c733650003
[junit4:junit4]   2> 1511 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650004 type:create cxid:0xe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 1511 T1207 oaz.ZooKeeper.close Session: 0x13ac1c733650003 closed
[junit4:junit4]   2> 1512 T1197 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60156 which had sessionid 0x13ac1c733650003
[junit4:junit4]   2> 1512 T1209 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1512 T1207 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:38243/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@e1a378
[junit4:junit4]   2> 1512 T1214 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:38243
[junit4:junit4]   2> 1512 T1207 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1513 T1214 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:38243, initiating session
[junit4:junit4]   2> 1513 T1197 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60158
[junit4:junit4]   2> 1513 T1197 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60158
[junit4:junit4]   2> 1514 T1214 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:38243, sessionid = 0x13ac1c733650005, negotiated timeout = 10000
[junit4:junit4]   2> 1514 T1199 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c733650005 with negotiated timeout 10000 for client /127.0.0.1:60158
[junit4:junit4]   2> 1514 T1215 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e1a378 name:ZooKeeperConnection Watcher:127.0.0.1:38243/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1515 T1207 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1516 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1517 T1207 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1518 T1207 oasc.Overseer.start Overseer (id=88596203992055813-127.0.0.1:38243_solr-n_0000000002) starting
[junit4:junit4]   2> 1519 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650005 type:create cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1519 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1520 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650005 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1520 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650005 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1521 T1217 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1522 T1216 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1523 T1216 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1523 T1216 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"state1",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 1523 T1216 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 1524 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650005 type:create cxid:0x18 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 1527 T1211 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 1738 T1194 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 1740 T1194 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88596203909808132-127.0.0.1:59194_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 1808 T1181 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 2013 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2019 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650004 type:delete cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 2019 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 2022 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650004 type:create cxid:0x28 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2023 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650004 type:create cxid:0x2a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2024 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2026 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650004 type:create cxid:0x31 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2028 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:38243/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@17ff48b
[junit4:junit4]   2> 2029 T1216 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2030 T1218 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:38243
[junit4:junit4]   2> 2030 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2030 T1218 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:38243, initiating session
[junit4:junit4]   2> 2030 T1197 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:60160
[junit4:junit4]   2> 2031 T1197 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:60160
[junit4:junit4]   2> 2032 T1199 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c733650006 with negotiated timeout 10000 for client /127.0.0.1:60160
[junit4:junit4]   2> 2032 T1218 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:38243, sessionid = 0x13ac1c733650006, negotiated timeout = 10000
[junit4:junit4]   2> 2032 T1219 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17ff48b name:ZooKeeperConnection Watcher:127.0.0.1:38243/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2033 T1216 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"state2",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 2032 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2033 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2035 T1219 oaz.ClientCnxn$EventThread.processEvent SEVERE Error while calling watcher  java.lang.NullPointerException
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:189)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 2035 T1211 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2035 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4:junit4]   2> 2037 T1211 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2038 T1219 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2038 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650004 type:create cxid:0x39 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2038 T1211 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 2039 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2041 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650004 type:create cxid:0x41 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2042 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2043 T1200 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c733650006 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2247 T1191 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 2248 T1191 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2248 T1191 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 2248 T1191 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 2249 T1185 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 2249 T1185 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 2512 T1213 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 2512 T1212 oasc.Overseer$ClusterStateUpdater.run WARNING Solr cannot talk to ZK
[junit4:junit4]   2> 2540 T1216 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2541 T1216 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"state1",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 2544 T1216 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "node_name":"node2",
[junit4:junit4]   2> 	  "state":"state2",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node2/solr/",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core":"core4",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 2546 T1211 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 62061 T1200 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c733650004
[junit4:junit4]   2> 62061 T1211 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 62062 T1211 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 62062 T1219 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 62062 T1211 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 62062 T1197 oazs.NIOServerCnxn.doIO WARNING EndOfStreamException: Unable to read additional data from client sessionid 0x13ac1c733650004, likely client has closed socket
[junit4:junit4]   2> 62062 T1178 oaz.ZooKeeper.close Session: 0x13ac1c733650004 closed
[junit4:junit4]   2> 62063 T1211 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 62063 T1197 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60157 which had sessionid 0x13ac1c733650004
[junit4:junit4]   2> 62064 T1211 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 62067 T1211 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 62067 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62170 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62272 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62374 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62476 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62578 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62680 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62783 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62885 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 62987 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63089 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63192 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63295 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63397 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63500 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63602 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63704 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63806 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 63909 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64011 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64113 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64216 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64318 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64420 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64523 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64626 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64728 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64831 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 64933 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65036 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65138 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65240 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65342 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65444 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65545 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65646 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65748 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65850 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 65952 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66054 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66155 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66256 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66358 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66459 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66561 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66663 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66764 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66866 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 66969 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67070 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67172 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67274 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67376 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67479 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67581 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67684 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67787 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67889 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 67991 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68092 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68194 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68295 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68397 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68498 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68600 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68702 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68803 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 68905 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69007 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69109 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69211 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69313 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69415 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69517 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69619 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69722 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69824 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 69926 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70029 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70135 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70238 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70340 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70443 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70545 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70647 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70749 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70851 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 70954 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71056 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71158 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71261 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71363 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71465 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71567 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71670 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71773 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71876 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 71978 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72081 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72183 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72286 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72388 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72490 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72593 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72695 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72797 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 72900 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73002 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73105 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73207 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73308 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73410 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73513 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73614 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73717 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73819 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73922 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74024 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74127 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74229 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74331 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74432 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74534 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74636 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74738 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74841 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 74944 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75047 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75149 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75252 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75354 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75456 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75559 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75662 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75764 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75866 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75968 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76071 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76174 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76276 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76379 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76481 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76583 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76685 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76787 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76889 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76992 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77094 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77196 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77299 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77401 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77504 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77607 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77709 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77811 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77913 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78015 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78117 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78219 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78322 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78425 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78527 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78629 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78731 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78834 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78937 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79039 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79141 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79244 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79346 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79448 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79550 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79652 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79755 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79857 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79959 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80061 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80163 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80265 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80367 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80470 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80572 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80674 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80777 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80879 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80982 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81084 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81186 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81288 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81390 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81492 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81594 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81697 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81799 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81902 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82004 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82105 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82207 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82308 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82410 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82604 T1200 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c733650005
[junit4:junit4]   2> 82605 T1207 oaz.ZooKeeper.close Session: 0x13ac1c733650005 closed
[junit4:junit4]   2> 82605 T1215 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82605 T1197 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60158 which had sessionid 0x13ac1c733650005
[junit4:junit4]   2> 82607 T1200 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c733650006
[junit4:junit4]   2> 82607 T1219 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 82607 T1197 oazs.NIOServerCnxn.doIO WARNING EndOfStreamException: Unable to read additional data from client sessionid 0x13ac1c733650006, likely client has closed socket
[junit4:junit4]   2> 82608 T1219 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82607 T1178 oaz.ZooKeeper.close Session: 0x13ac1c733650006 closed
[junit4:junit4]   2> 82608 T1197 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60160 which had sessionid 0x13ac1c733650006
[junit4:junit4]   2> 82608 T1200 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c733650000
[junit4:junit4]   2> 82612 T1202 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82612 T1197 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:60153 which had sessionid 0x13ac1c733650000
[junit4:junit4]   2> 82612 T1178 oaz.ZooKeeper.close Session: 0x13ac1c733650000 closed
[junit4:junit4]   2> 82612 T1200 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 82612 T1199 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 82613 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 82615 T1197 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 82615 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 82616 T1178 oas.SolrTestCaseJ4.tearDown ###Ending testShardLeaderChange
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testShardLeaderChange -Dtests.seed=B8C7B18B86CE8EE3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=uk -Dtests.timezone=America/St_Vincent -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 81.4s J1 | OverseerTest.testShardLeaderChange <<<
[junit4:junit4]    > Throwable #1: org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([B8C7B18B86CE8EE3:6694367C9C567B12]:0)
[junit4:junit4]    > 	at org.junit.Assert.assertEquals(Assert.java:125)
[junit4:junit4]    > 	at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:493)
[junit4:junit4]    > 	at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:652)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:474)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 82672 T1178 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4:junit4]   2> 82672 T1178 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 82673 T1220 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 82674 T1220 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 82675 T1220 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351870787193/zookeeper/server1/data/version-2/snapshot.2e
[junit4:junit4]   2> 82679 T1220 oazsp.FileTxnSnapLog.save Snapshotting: 91
[junit4:junit4]   2> 82773 T1178 oasc.ZkTestServer.run start zk server on port:60873
[junit4:junit4]   2> 82774 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:60873/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@c39715
[junit4:junit4]   2> 82775 T1225 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:60873
[junit4:junit4]   2> 82775 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 82776 T1225 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:60873, initiating session
[junit4:junit4]   2> 82776 T1221 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35112
[junit4:junit4]   2> 82776 T1221 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35112
[junit4:junit4]   2> 82776 T1223 oazsp.FileTxnLog.append Creating new log file: log.92
[junit4:junit4]   2> 82778 T1223 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c8716f0000 with negotiated timeout 10000 for client /127.0.0.1:35112
[junit4:junit4]   2> 82778 T1225 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:60873, sessionid = 0x13ac1c8716f0000, negotiated timeout = 10000
[junit4:junit4]   2> 82778 T1226 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c39715 name:ZooKeeperConnection Watcher:127.0.0.1:60873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 82778 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 82779 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:60873 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1699a1
[junit4:junit4]   2> 82779 T1227 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:60873
[junit4:junit4]   2> 82779 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 82780 T1227 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:60873, initiating session
[junit4:junit4]   2> 82780 T1221 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35113
[junit4:junit4]   2> 82780 T1221 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35113
[junit4:junit4]   2> 82792 T1223 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c8716f0001 with negotiated timeout 10000 for client /127.0.0.1:35113
[junit4:junit4]   2> 82792 T1227 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:60873, sessionid = 0x13ac1c8716f0001, negotiated timeout = 10000
[junit4:junit4]   2> 82792 T1228 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1699a1 name:ZooKeeperConnection Watcher:127.0.0.1:60873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 82793 T1216 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 82793 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 82793 T1216 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88596203992055813-127.0.0.1:38243_solr-n_0000000002) am no longer a leader.
[junit4:junit4]   2> 82801 T1224 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c8716f0001
[junit4:junit4]   2> 82802 T1228 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82802 T1221 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35113 which had sessionid 0x13ac1c8716f0001
[junit4:junit4]   2> 82802 T1178 oaz.ZooKeeper.close Session: 0x13ac1c8716f0001 closed
[junit4:junit4]   2> 82802 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:60873 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@19ef78
[junit4:junit4]   2> 82803 T1229 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:60873
[junit4:junit4]   2> 82803 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 82803 T1229 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:60873, initiating session
[junit4:junit4]   2> 82804 T1221 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35114
[junit4:junit4]   2> 82804 T1221 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35114
[junit4:junit4]   2> 82804 T1223 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c8716f0002 with negotiated timeout 10000 for client /127.0.0.1:35114
[junit4:junit4]   2> 82804 T1229 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:60873, sessionid = 0x13ac1c8716f0002, negotiated timeout = 10000
[junit4:junit4]   2> 82805 T1230 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19ef78 name:ZooKeeperConnection Watcher:127.0.0.1:60873 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 82805 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 82805 T1178 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 82807 T1224 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c8716f0002
[junit4:junit4]   2> 82807 T1221 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35114 which had sessionid 0x13ac1c8716f0002
[junit4:junit4]   2> 82807 T1178 oaz.ZooKeeper.close Session: 0x13ac1c8716f0002 closed
[junit4:junit4]   2> 82808 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 82808 T1230 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82808 T1198 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 82809 T1178 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 82810 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 82812 T1224 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c8716f0000 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 82813 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:60873/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@36290
[junit4:junit4]   2> 82814 T1231 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:60873
[junit4:junit4]   2> 82814 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 82815 T1231 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:60873, initiating session
[junit4:junit4]   2> 82815 T1221 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:35115
[junit4:junit4]   2> 82815 T1221 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:35115
[junit4:junit4]   2> 82816 T1223 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c8716f0003 with negotiated timeout 10000 for client /127.0.0.1:35115
[junit4:junit4]   2> 82816 T1231 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:60873, sessionid = 0x13ac1c8716f0003, negotiated timeout = 10000
[junit4:junit4]   2> 82816 T1232 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36290 name:ZooKeeperConnection Watcher:127.0.0.1:60873/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 82816 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 82817 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 82819 T1224 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c8716f0003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 82820 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 82821 T1178 oasc.Overseer.start Overseer (id=88596209327865859-127.0.0.1:60873_solr-n_0000000000) starting
[junit4:junit4]   2> 82822 T1224 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c8716f0003 type:create cxid:0xe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 82834 T1224 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c8716f0003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 82835 T1224 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c8716f0003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 82836 T1224 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c8716f0003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 82837 T1234 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 82837 T1224 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c8716f0000 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 82838 T1224 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c8716f0000 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 82838 T1233 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82839 T1233 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4:junit4]   2> 82839 T1233 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "shard":"s1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core1"}
[junit4:junit4]   2> 82840 T1226 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 82842 T1233 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "shard":"s1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core2"}
[junit4:junit4]   2> 82843 T1226 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 82843 T1233 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 82844 T1233 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82845 T1233 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "roles":"",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "shard":"s1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"core3"}
[junit4:junit4]   2> 82847 T1226 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 82940 T1224 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c8716f0003
[junit4:junit4]   2> 82952 T1232 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82952 T1221 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35115 which had sessionid 0x13ac1c8716f0003
[junit4:junit4]   2> 82952 T1178 oaz.ZooKeeper.close Session: 0x13ac1c8716f0003 closed
[junit4:junit4]   2> 82953 T1224 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c8716f0000
[junit4:junit4]   2> 82971 T1226 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 82972 T1221 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:35112 which had sessionid 0x13ac1c8716f0000
[junit4:junit4]   2> 82971 T1178 oaz.ZooKeeper.close Session: 0x13ac1c8716f0000 closed
[junit4:junit4]   2> 82972 T1224 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 82972 T1223 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 82973 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 82975 T1221 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 82976 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 82976 T1178 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4:junit4]   2> 82983 T1178 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
[junit4:junit4]   2> 82983 T1178 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 82984 T1235 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 82984 T1235 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 82985 T1235 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351870787193/zookeeper/server1/data/version-2/snapshot.91
[junit4:junit4]   2> 82989 T1235 oazsp.FileTxnSnapLog.save Snapshotting: c6
[junit4:junit4]   2> 83084 T1178 oasc.ZkTestServer.run start zk server on port:47076
[junit4:junit4]   2> 83085 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47076/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1206689
[junit4:junit4]   2> 83086 T1240 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47076
[junit4:junit4]   2> 83086 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 83086 T1240 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47076, initiating session
[junit4:junit4]   2> 83086 T1236 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46304
[junit4:junit4]   2> 83087 T1236 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46304
[junit4:junit4]   2> 83087 T1238 oazsp.FileTxnLog.append Creating new log file: log.c7
[junit4:junit4]   2> 83348 T1233 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 83349 T1233 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88596209327865859-127.0.0.1:60873_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 83613 T1238 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c872a50000 with negotiated timeout 10000 for client /127.0.0.1:46304
[junit4:junit4]   2> 83613 T1240 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47076, sessionid = 0x13ac1c872a50000, negotiated timeout = 10000
[junit4:junit4]   2> 83613 T1241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1206689 name:ZooKeeperConnection Watcher:127.0.0.1:47076/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 83614 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 83615 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47076 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1d24eac
[junit4:junit4]   2> 83616 T1242 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47076
[junit4:junit4]   2> 83617 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 83618 T1242 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47076, initiating session
[junit4:junit4]   2> 83618 T1236 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46305
[junit4:junit4]   2> 83618 T1236 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46305
[junit4:junit4]   2> 83620 T1238 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c872a50001 with negotiated timeout 10000 for client /127.0.0.1:46305
[junit4:junit4]   2> 83620 T1242 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47076, sessionid = 0x13ac1c872a50001, negotiated timeout = 10000
[junit4:junit4]   2> 83620 T1243 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d24eac name:ZooKeeperConnection Watcher:127.0.0.1:47076 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 83621 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 83629 T1239 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c872a50001
[junit4:junit4]   2> 83630 T1236 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46305 which had sessionid 0x13ac1c872a50001
[junit4:junit4]   2> 83630 T1178 oaz.ZooKeeper.close Session: 0x13ac1c872a50001 closed
[junit4:junit4]   2> 83630 T1243 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 83630 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47076 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@f8bb97
[junit4:junit4]   2> 83631 T1244 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47076
[junit4:junit4]   2> 83631 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 83632 T1244 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47076, initiating session
[junit4:junit4]   2> 83632 T1236 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46306
[junit4:junit4]   2> 83632 T1236 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46306
[junit4:junit4]   2> 83633 T1244 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47076, sessionid = 0x13ac1c872a50002, negotiated timeout = 10000
[junit4:junit4]   2> 83632 T1238 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c872a50002 with negotiated timeout 10000 for client /127.0.0.1:46306
[junit4:junit4]   2> 83633 T1245 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8bb97 name:ZooKeeperConnection Watcher:127.0.0.1:47076 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 83633 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 83634 T1178 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 83643 T1239 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c872a50002
[junit4:junit4]   2> 83643 T1245 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 83643 T1236 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46306 which had sessionid 0x13ac1c872a50002
[junit4:junit4]   2> 83643 T1178 oaz.ZooKeeper.close Session: 0x13ac1c872a50002 closed
[junit4:junit4]   2> 83644 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 83645 T1178 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 83646 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 83647 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47076/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1ff5c0
[junit4:junit4]   2> 83648 T1246 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47076
[junit4:junit4]   2> 83648 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 83648 T1246 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47076, initiating session
[junit4:junit4]   2> 83648 T1236 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46307
[junit4:junit4]   2> 83649 T1236 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46307
[junit4:junit4]   2> 83649 T1238 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c872a50003 with negotiated timeout 10000 for client /127.0.0.1:46307
[junit4:junit4]   2> 83649 T1246 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47076, sessionid = 0x13ac1c872a50003, negotiated timeout = 10000
[junit4:junit4]   2> 83650 T1247 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ff5c0 name:ZooKeeperConnection Watcher:127.0.0.1:47076/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 83650 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 83651 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 83652 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 83653 T1247 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 83653 T1241 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 83653 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47076/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@40db52
[junit4:junit4]   2> 83654 T1248 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47076
[junit4:junit4]   2> 83654 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 83654 T1248 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47076, initiating session
[junit4:junit4]   2> 83654 T1236 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46308
[junit4:junit4]   2> 83655 T1236 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46308
[junit4:junit4]   2> 83655 T1238 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c872a50004 with negotiated timeout 10000 for client /127.0.0.1:46308
[junit4:junit4]   2> 83655 T1248 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47076, sessionid = 0x13ac1c872a50004, negotiated timeout = 10000
[junit4:junit4]   2> 83656 T1249 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40db52 name:ZooKeeperConnection Watcher:127.0.0.1:47076/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 83656 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 83656 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 83659 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 83659 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 83660 T1178 oasc.Overseer.start Overseer (id=88596209348182020-127.0.0.1:47076_solr-n_0000000000) starting
[junit4:junit4]   2> 83661 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 83662 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 83662 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 83663 T1251 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 83665 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 83665 T1250 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 83665 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 83808 T1222 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 84168 T1250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84169 T1250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 84169 T1250 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 84169 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50004 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 84252 T1247 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 84252 T1241 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 84667 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 84927 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50003 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 84928 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 84934 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50003 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 84936 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84942 T1241 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 84942 T1247 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 84943 T1239 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c872a50003
[junit4:junit4]   2> 84945 T1236 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46307 which had sessionid 0x13ac1c872a50003
[junit4:junit4]   2> 84945 T1178 oaz.ZooKeeper.close Session: 0x13ac1c872a50003 closed
[junit4:junit4]   2> 84946 T1241 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 84946 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:47076/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@46a453
[junit4:junit4]   2> 84948 T1252 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:47076
[junit4:junit4]   2> 84948 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 84949 T1236 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:46309
[junit4:junit4]   2> 84949 T1252 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:47076, initiating session
[junit4:junit4]   2> 84950 T1236 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:46309
[junit4:junit4]   2> 84952 T1252 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:47076, sessionid = 0x13ac1c872a50005, negotiated timeout = 10000
[junit4:junit4]   2> 84952 T1238 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c872a50005 with negotiated timeout 10000 for client /127.0.0.1:46309
[junit4:junit4]   2> 84952 T1253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46a453 name:ZooKeeperConnection Watcher:127.0.0.1:47076/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 84953 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 84954 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 84956 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 84962 T1253 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 84962 T1241 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 84963 T1241 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 84965 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 84967 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 84970 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50005 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 84972 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50005 type:delete cxid:0x14 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard1 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard1
[junit4:junit4]   2> 84973 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 84976 T1239 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c872a50005 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85424 T1250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85434 T1250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 85441 T1253 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 85441 T1241 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 85441 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85443 T1239 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c872a50004
[junit4:junit4]   2> 85451 T1249 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 85451 T1178 oaz.ZooKeeper.close Session: 0x13ac1c872a50004 closed
[junit4:junit4]   2> 85451 T1236 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46308 which had sessionid 0x13ac1c872a50004
[junit4:junit4]   2> 85453 T1241 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 85453 T1253 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 85453 T1239 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c872a50005
[junit4:junit4]   2> 85454 T1236 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46309 which had sessionid 0x13ac1c872a50005
[junit4:junit4]   2> 85454 T1178 oaz.ZooKeeper.close Session: 0x13ac1c872a50005 closed
[junit4:junit4]   2> 85455 T1239 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c872a50000
[junit4:junit4]   2> 85455 T1236 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:46304 which had sessionid 0x13ac1c872a50000
[junit4:junit4]   2> 85455 T1178 oaz.ZooKeeper.close Session: 0x13ac1c872a50000 closed
[junit4:junit4]   2> 85456 T1239 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 85456 T1238 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 85456 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 85458 T1236 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 85459 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 85459 T1178 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
[junit4:junit4]   2> 85466 T1178 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
[junit4:junit4]   2> 85466 T1178 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 85466 T1254 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 85467 T1254 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 85467 T1254 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351870787193/zookeeper/server1/data/version-2/snapshot.c6
[junit4:junit4]   2> 85470 T1254 oazsp.FileTxnSnapLog.save Snapshotting: 110
[junit4:junit4]   2> 85566 T1178 oasc.ZkTestServer.run start zk server on port:36109
[junit4:junit4]   2> 85567 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:36109/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@cea838
[junit4:junit4]   2> 85568 T1259 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:36109
[junit4:junit4]   2> 85568 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 85568 T1259 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:36109, initiating session
[junit4:junit4]   2> 85568 T1255 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:37076
[junit4:junit4]   2> 85569 T1255 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:37076
[junit4:junit4]   2> 85569 T1257 oazsp.FileTxnLog.append Creating new log file: log.111
[junit4:junit4]   2> 85591 T1259 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:36109, sessionid = 0x13ac1c87c570000, negotiated timeout = 10000
[junit4:junit4]   2> 85591 T1257 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c87c570000 with negotiated timeout 10000 for client /127.0.0.1:37076
[junit4:junit4]   2> 85592 T1260 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cea838 name:ZooKeeperConnection Watcher:127.0.0.1:36109/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 85592 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 85593 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:36109 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@ef5351
[junit4:junit4]   2> 85594 T1261 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:36109
[junit4:junit4]   2> 85594 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 85594 T1261 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:36109, initiating session
[junit4:junit4]   2> 85594 T1255 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:37077
[junit4:junit4]   2> 85595 T1255 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:37077
[junit4:junit4]   2> 85597 T1261 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:36109, sessionid = 0x13ac1c87c570001, negotiated timeout = 10000
[junit4:junit4]   2> 85597 T1257 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c87c570001 with negotiated timeout 10000 for client /127.0.0.1:37077
[junit4:junit4]   2> 85597 T1262 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ef5351 name:ZooKeeperConnection Watcher:127.0.0.1:36109 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 85597 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 85622 T1258 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c87c570001
[junit4:junit4]   2> 85623 T1262 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 85623 T1255 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37077 which had sessionid 0x13ac1c87c570001
[junit4:junit4]   2> 85623 T1178 oaz.ZooKeeper.close Session: 0x13ac1c87c570001 closed
[junit4:junit4]   2> 85624 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:36109 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@323fe
[junit4:junit4]   2> 85624 T1263 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:36109
[junit4:junit4]   2> 85624 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 85625 T1263 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:36109, initiating session
[junit4:junit4]   2> 85625 T1255 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:37078
[junit4:junit4]   2> 85626 T1255 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:37078
[junit4:junit4]   2> 85626 T1263 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:36109, sessionid = 0x13ac1c87c570002, negotiated timeout = 10000
[junit4:junit4]   2> 85626 T1257 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c87c570002 with negotiated timeout 10000 for client /127.0.0.1:37078
[junit4:junit4]   2> 85627 T1264 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@323fe name:ZooKeeperConnection Watcher:127.0.0.1:36109 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 85627 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 85628 T1178 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 85629 T1258 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c87c570002
[junit4:junit4]   2> 85630 T1255 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37078 which had sessionid 0x13ac1c87c570002
[junit4:junit4]   2> 85630 T1178 oaz.ZooKeeper.close Session: 0x13ac1c87c570002 closed
[junit4:junit4]   2> 85630 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 85632 T1264 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 85634 T1178 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 85636 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 85637 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:36109/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@11dfb35
[junit4:junit4]   2> 85637 T1265 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:36109
[junit4:junit4]   2> 85637 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 85638 T1265 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:36109, initiating session
[junit4:junit4]   2> 85638 T1255 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:37079
[junit4:junit4]   2> 85638 T1255 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:37079
[junit4:junit4]   2> 85639 T1257 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c87c570003 with negotiated timeout 10000 for client /127.0.0.1:37079
[junit4:junit4]   2> 85639 T1265 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:36109, sessionid = 0x13ac1c87c570003, negotiated timeout = 10000
[junit4:junit4]   2> 85639 T1266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11dfb35 name:ZooKeeperConnection Watcher:127.0.0.1:36109/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 85640 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 85640 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 85641 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 85643 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:36109/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1e3901e
[junit4:junit4]   2> 85644 T1260 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 85644 T1266 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 85644 T1267 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:36109
[junit4:junit4]   2> 85644 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 85645 T1267 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:36109, initiating session
[junit4:junit4]   2> 85645 T1255 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:37080
[junit4:junit4]   2> 85645 T1255 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:37080
[junit4:junit4]   2> 85646 T1267 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:36109, sessionid = 0x13ac1c87c570004, negotiated timeout = 10000
[junit4:junit4]   2> 85646 T1257 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c87c570004 with negotiated timeout 10000 for client /127.0.0.1:37080
[junit4:junit4]   2> 85646 T1268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e3901e name:ZooKeeperConnection Watcher:127.0.0.1:36109/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 85647 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 85648 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 85650 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 85651 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 85653 T1178 oasc.Overseer.start Overseer (id=88596209510842372-127.0.0.1:36109_solr-n_0000000000) starting
[junit4:junit4]   2> 85655 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85656 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85657 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 85658 T1270 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 85658 T1269 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 85808 T1237 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 85944 T1247 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 85944 T1247 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 85944 T1247 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 86443 T1250 oasc.Overseer$ClusterStateUpdater.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 86454 T1253 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 86454 T1253 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 86454 T1253 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 86455 T1241 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 86455 T1241 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 86659 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 86660 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 87175 T1269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87176 T1269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 87177 T1269 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 87177 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570004 type:create cxid:0x26 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 87206 T1266 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 87206 T1260 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 87673 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 87677 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:delete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 87678 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 87680 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87681 T1178 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87682 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 87683 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 87684 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0x2e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 87708 T1269 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87714 T1269 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 87723 T1260 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 87723 T1266 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 87774 T1258 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c87c570004
[junit4:junit4]   2> 87776 T1268 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 87776 T1255 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37080 which had sessionid 0x13ac1c87c570004
[junit4:junit4]   2> 87776 T1178 oaz.ZooKeeper.close Session: 0x13ac1c87c570004 closed
[junit4:junit4]   2> 88225 T1269 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 88226 T1269 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88596209510842372-127.0.0.1:36109_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 88777 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0x34 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89271 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 89275 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0x3b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 89277 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:36109/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@18fc059
[junit4:junit4]   2> 89279 T1271 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:36109
[junit4:junit4]   2> 89279 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 89280 T1271 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:36109, initiating session
[junit4:junit4]   2> 89280 T1255 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:37081
[junit4:junit4]   2> 89281 T1255 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:37081
[junit4:junit4]   2> 89282 T1271 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:36109, sessionid = 0x13ac1c87c570005, negotiated timeout = 10000
[junit4:junit4]   2> 89282 T1257 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c87c570005 with negotiated timeout 10000 for client /127.0.0.1:37081
[junit4:junit4]   2> 89283 T1272 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18fc059 name:ZooKeeperConnection Watcher:127.0.0.1:36109/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 89283 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 89292 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 89294 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 89296 T1178 oasc.Overseer.start Overseer (id=88596209510842373-127.0.0.1:36109_solr-n_0000000002) starting
[junit4:junit4]   2> 89298 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570005 type:create cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89299 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89300 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570005 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89301 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570005 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89303 T1274 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 89304 T1273 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 89306 T1273 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89307 T1273 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1"}
[junit4:junit4]   2> 89313 T1260 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 89313 T1266 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 89364 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0x41 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89365 T1178 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 89367 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0x48 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 89816 T1273 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 89852 T1266 oasc.SolrException.log SEVERE Failed setting watch:org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election/88596209510842371-node1_core1-n_0000000000
[junit4:junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 		at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:116)
[junit4:junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 	
[junit4:junit4]   2> 89892 T1266 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 89952 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570005 type:delete cxid:0x35 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = Directory not empty for /solr/collections/collection1/leaders
[junit4:junit4]   2> 89992 T1258 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c87c570003 type:create cxid:0x56 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 89992 T1273 oasc.SolrException.log SEVERE Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leaders
[junit4:junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:116)
[junit4:junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 		at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:734)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:144)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:141)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:141)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:601)
[junit4:junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:596)
[junit4:junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:421)
[junit4:junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:168)
[junit4:junit4]   2> 		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:129)
[junit4:junit4]   2> 		at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 	
[junit4:junit4]   2> 90222 T1266 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 90223 T1273 oasc.Overseer$ClusterStateUpdater.setShardLeader SEVERE Could not mark shard leader for non existing collection:collection1
[junit4:junit4]   2> 90224 T1266 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 90225 T1266 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 90225 T1260 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 90742 T1258 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c87c570003
[junit4:junit4]   2> 90742 T1260 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 90742 T1266 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 90742 T1260 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 90742 T1266 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 90762 T1255 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37079 which had sessionid 0x13ac1c87c570003
[junit4:junit4]   2> 90762 T1178 oaz.ZooKeeper.close Session: 0x13ac1c87c570003 closed
[junit4:junit4]   2> 90763 T1258 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c87c570005
[junit4:junit4]   2> 90782 T1272 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 90782 T1255 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37081 which had sessionid 0x13ac1c87c570005
[junit4:junit4]   2> 90782 T1178 oaz.ZooKeeper.close Session: 0x13ac1c87c570005 closed
[junit4:junit4]   2> 90782 T1258 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c87c570000
[junit4:junit4]   2> 90783 T1260 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 90783 T1255 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:37076 which had sessionid 0x13ac1c87c570000
[junit4:junit4]   2> 90783 T1178 oaz.ZooKeeper.close Session: 0x13ac1c87c570000 closed
[junit4:junit4]   2> 90784 T1258 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 90784 T1257 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 90784 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 90786 T1255 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 90786 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 90787 T1178 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4:junit4]   2> 90791 T1178 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
[junit4:junit4]   2> 90792 T1178 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 90792 T1275 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 90793 T1275 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 90794 T1275 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351870787193/zookeeper/server1/data/version-2/snapshot.110
[junit4:junit4]   2> 90798 T1275 oazsp.FileTxnSnapLog.save Snapshotting: 183
[junit4:junit4]   2> 90808 T1256 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 90892 T1178 oasc.ZkTestServer.run start zk server on port:46723
[junit4:junit4]   2> 90893 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@167e06b
[junit4:junit4]   2> 90894 T1280 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 90894 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90895 T1280 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 90895 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57536
[junit4:junit4]   2> 90895 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57536
[junit4:junit4]   2> 90896 T1278 oazsp.FileTxnLog.append Creating new log file: log.184
[junit4:junit4]   2> 90897 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270000 with negotiated timeout 10000 for client /127.0.0.1:57536
[junit4:junit4]   2> 90897 T1280 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270000, negotiated timeout = 10000
[junit4:junit4]   2> 90898 T1281 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@167e06b name:ZooKeeperConnection Watcher:127.0.0.1:46723 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90898 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90918 T1279 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c891270000
[junit4:junit4]   2> 90919 T1281 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 90919 T1276 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57536 which had sessionid 0x13ac1c891270000
[junit4:junit4]   2> 90919 T1178 oaz.ZooKeeper.close Session: 0x13ac1c891270000 closed
[junit4:junit4]   2> 90920 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@16ac426
[junit4:junit4]   2> 90920 T1282 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 90920 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90921 T1282 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 90921 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57537
[junit4:junit4]   2> 90921 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57537
[junit4:junit4]   2> 90922 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270001 with negotiated timeout 10000 for client /127.0.0.1:57537
[junit4:junit4]   2> 90922 T1282 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270001, negotiated timeout = 10000
[junit4:junit4]   2> 90922 T1283 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16ac426 name:ZooKeeperConnection Watcher:127.0.0.1:46723 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90923 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90923 T1178 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 90927 T1279 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c891270001
[junit4:junit4]   2> 90928 T1283 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 90928 T1276 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:57537 which had sessionid 0x13ac1c891270001
[junit4:junit4]   2> 90928 T1178 oaz.ZooKeeper.close Session: 0x13ac1c891270001 closed
[junit4:junit4]   2> 90928 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@3852bb
[junit4:junit4]   2> 90929 T1284 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 90929 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90930 T1284 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 90930 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57538
[junit4:junit4]   2> 90930 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57538
[junit4:junit4]   2> 90931 T1284 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270002, negotiated timeout = 10000
[junit4:junit4]   2> 90931 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270002 with negotiated timeout 10000 for client /127.0.0.1:57538
[junit4:junit4]   2> 90931 T1285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3852bb name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90932 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90932 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 90942 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@43d679
[junit4:junit4]   2> 90943 T1286 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 90943 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90943 T1286 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 90943 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57539
[junit4:junit4]   2> 90944 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57539
[junit4:junit4]   2> 90944 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270003 with negotiated timeout 10000 for client /127.0.0.1:57539
[junit4:junit4]   2> 90945 T1286 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270003, negotiated timeout = 10000
[junit4:junit4]   2> 90945 T1287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43d679 name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90945 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90946 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 90948 T1279 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c891270003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 90949 T1178 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 90952 T1178 oasc.Overseer.start Overseer (id=88596209860018179-127.0.0.1:46723_solr-n_0000000000) starting
[junit4:junit4]   2> 90954 T1279 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c891270003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 90955 T1279 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c891270003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 90955 T1279 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac1c891270003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 90956 T1289 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 90956 T1178 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 90957 T1288 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 90959 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 90960 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1b5a00b
[junit4:junit4]   2> 90961 T1290 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 90961 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90961 T1290 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 90962 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57540
[junit4:junit4]   2> 90962 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57540
[junit4:junit4]   2> 90962 T1290 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270004, negotiated timeout = 10000
[junit4:junit4]   2> 90962 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270004 with negotiated timeout 10000 for client /127.0.0.1:57540
[junit4:junit4]   2> 90963 T1291 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b5a00b name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90963 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90964 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 90964 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node0
[junit4:junit4]   2> 90966 T1285 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 90967 T1291 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 90967 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1a72333
[junit4:junit4]   2> 90967 T1292 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 90967 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90968 T1292 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 90968 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57541
[junit4:junit4]   2> 90968 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57541
[junit4:junit4]   2> 90969 T1292 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270005, negotiated timeout = 10000
[junit4:junit4]   2> 90969 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270005 with negotiated timeout 10000 for client /127.0.0.1:57541
[junit4:junit4]   2> 90969 T1293 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a72333 name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90970 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90970 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 90972 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 90973 T1285 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 90974 T1293 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 90973 T1291 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 90974 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1291743
[junit4:junit4]   2> 90975 T1294 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 90975 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90975 T1294 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 90976 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57542
[junit4:junit4]   2> 90976 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57542
[junit4:junit4]   2> 90977 T1294 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270006, negotiated timeout = 10000
[junit4:junit4]   2> 90977 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270006 with negotiated timeout 10000 for client /127.0.0.1:57542
[junit4:junit4]   2> 90977 T1295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1291743 name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90977 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90978 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 90979 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4:junit4]   2> 90981 T1293 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 90981 T1291 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 90981 T1285 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 90981 T1295 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 90982 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@e60a86
[junit4:junit4]   2> 90983 T1296 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 90983 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90983 T1296 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 90983 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57543
[junit4:junit4]   2> 90984 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57543
[junit4:junit4]   2> 90992 T1296 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270007, negotiated timeout = 10000
[junit4:junit4]   2> 90992 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270007 with negotiated timeout 10000 for client /127.0.0.1:57543
[junit4:junit4]   2> 90992 T1297 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e60a86 name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90992 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90993 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 90994 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node3
[junit4:junit4]   2> 90995 T1291 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 90995 T1293 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 90995 T1285 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 90995 T1295 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 90996 T1297 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 90996 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1f10f01
[junit4:junit4]   2> 90997 T1298 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 90997 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90998 T1298 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 90998 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57544
[junit4:junit4]   2> 90998 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57544
[junit4:junit4]   2> 90999 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270008 with negotiated timeout 10000 for client /127.0.0.1:57544
[junit4:junit4]   2> 90999 T1298 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270008, negotiated timeout = 10000
[junit4:junit4]   2> 90999 T1299 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f10f01 name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90999 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 91000 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 91001 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node4
[junit4:junit4]   2> 91004 T1293 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 91004 T1291 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 91004 T1299 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 91004 T1295 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 91004 T1285 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 91004 T1297 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 91004 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1b73984
[junit4:junit4]   2> 91005 T1300 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 91005 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 91006 T1300 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 91006 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57545
[junit4:junit4]   2> 91006 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57545
[junit4:junit4]   2> 91012 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c891270009 with negotiated timeout 10000 for client /127.0.0.1:57545
[junit4:junit4]   2> 91012 T1300 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c891270009, negotiated timeout = 10000
[junit4:junit4]   2> 91012 T1301 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b73984 name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 91012 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 91013 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 91014 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node5
[junit4:junit4]   2> 91017 T1293 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 91017 T1295 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 91017 T1301 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 91017 T1299 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 91017 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1319bc2
[junit4:junit4]   2> 91017 T1297 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 91017 T1285 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 91017 T1291 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 91019 T1302 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 91019 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 91019 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57546
[junit4:junit4]   2> 91019 T1302 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 91020 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57546
[junit4:junit4]   2> 91020 T1302 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c89127000a, negotiated timeout = 10000
[junit4:junit4]   2> 91020 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c89127000a with negotiated timeout 10000 for client /127.0.0.1:57546
[junit4:junit4]   2> 91021 T1303 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1319bc2 name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 91021 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 91024 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 91027 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node6
[junit4:junit4]   2> 91029 T1293 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91029 T1301 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91031 T1295 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91031 T1303 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91031 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@458938
[junit4:junit4]   2> 91031 T1299 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91031 T1297 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91031 T1285 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91031 T1291 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91033 T1304 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 91033 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 91033 T1304 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 91033 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57547
[junit4:junit4]   2> 91034 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57547
[junit4:junit4]   2> 91042 T1304 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c89127000b, negotiated timeout = 10000
[junit4:junit4]   2> 91042 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c89127000b with negotiated timeout 10000 for client /127.0.0.1:57547
[junit4:junit4]   2> 91042 T1305 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@458938 name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 91043 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 91043 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 91044 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node7
[junit4:junit4]   2> 91046 T1293 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 91049 T1301 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 91049 T1303 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 91049 T1297 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 91050 T1178 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:46723/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@dd561e
[junit4:junit4]   2> 91049 T1285 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 91049 T1305 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 91049 T1291 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 91049 T1299 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 91050 T1306 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:46723
[junit4:junit4]   2> 91050 T1295 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 91050 T1178 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 91051 T1306 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:46723, initiating session
[junit4:junit4]   2> 91051 T1276 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:57548
[junit4:junit4]   2> 91052 T1276 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:57548
[junit4:junit4]   2> 91053 T1278 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac1c89127000c with negotiated timeout 10000 for client /127.0.0.1:57548
[junit4:junit4]   2> 91053 T1306 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:46723, sessionid = 0x13ac1c89127000c, negotiated timeout = 10000
[junit4:junit4]   2> 91054 T1307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@dd561e name:ZooKeeperConnection Watcher:127.0.0.1:46723/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 91054 T1178 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 91055 T1178 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 91056 T1178 oascc.SolrZkClient.makePath makePath: /live_nodes/node8
[junit4:junit4]   2> 91062 T1295 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4:junit4]   2> 91062 T1301 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4:junit4]   2> 91062 T1291 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4:junit4]   2> 91062 T1297 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4:junit4]   2> 91063 T1303 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4:junit4]   2> 91063 T1305 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4:junit4]   2> 91062 T1299 oascc.ZkStateReader$3.process Updating live nodes... (9)
[junit4:junit4]   2> 91062 T1293 oascc.ZkStateReader$3.proc

[...truncated too long message...]

nit4]   2> 104983 T1504 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac1c8b40a0003
[junit4:junit4]   2> 104992 T1512 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 104992 T1501 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45074 which had sessionid 0x13ac1c8b40a0003
[junit4:junit4]   2> 104992 T1178 oaz.ZooKeeper.close Session: 0x13ac1c8b40a0003 closed
[junit4:junit4]   2> 104993 T1504 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 104993 T1503 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 104995 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 104998 T1501 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 104999 T1178 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 105000 T1178 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4:junit4]   2> 105003 T1178 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> 105003 T1178 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 105433 T1513 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 105434 T1513 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88596210445320195-127.0.0.1:44424_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 105808 T1502 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 105972 T1516 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 105973 T1516 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 105973 T1516 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 105974 T1516 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 105975 T1516 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 105976 T1516 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 108004 T1178 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=uk, timezone=America/St_Vincent
[junit4:junit4]   2> NOTE: Linux 3.2.0-32-generic i386/Oracle Corporation 1.8.0-ea (32-bit)/cpus=8,threads=10,free=84344760,total=304390144
[junit4:junit4]   2> NOTE: All tests run in this JVM: [DistributedQueryElevationComponentTest, TestWriterPerf, TestPropInject, TestLRUCache, FileUtilsTest, TestFastWriter, UUIDFieldTest, ShowFileRequestHandlerTest, TestSolrCoreProperties, URLClassifyProcessorTest, PathHierarchyTokenizerFactoryTest, MultiTermTest, BasicFunctionalityTest, TestReload, TestJmxIntegration, TestDistributedSearch, TestReversedWildcardFilterFactory, SpellingQueryConverterTest, TestFunctionQuery, SolrIndexConfigTest, DirectUpdateHandlerTest, PolyFieldTest, PrimUtilsTest, TestRealTimeGet, TestSolrXMLSerializer, ReturnFieldsTest, TestUpdate, SimplePostToolTest, LeaderElectionTest, SignatureUpdateProcessorFactoryTest, TestUtils, QueryEqualityTest, TestSurroundQueryParser, TermVectorComponentDistributedTest, PingRequestHandlerTest, PrimitiveFieldTypeTest, ZkNodePropsTest, OutputWriterTest, TestFastOutputStream, SolrCoreTest, TestSolrIndexConfig, TestStressVersions, EchoParamsTest, TestRemoteStreaming, DirectSolrSpellCheckerTest, TermVectorComponentTest, FullSolrCloudDistribCmdsTest, TestSort, TestHashPartitioner, TestQueryUtils, ConvertedLegacyTest, PeerSyncTest, OverseerTest]
[junit4:junit4] Completed on J1 in 108.05s, 8 tests, 1 failure <<< FAILURES!

[...truncated 374 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:335: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:178: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1163: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 1 failure, 7 ignored (2 assumptions)

Total time: 25 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/jdk1.8.0-ea-b58 -client -XX:+UseSerialGC
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message