lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@sd-datasolutions.de>
Subject [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_07) - Build # 2028 - Failure!
Date Sat, 27 Oct 2012 21:22:41 GMT
Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-4.x-Linux/2028/
Java: 32bit/jdk1.7.0_07 -server -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.OverseerTest.testShardLeaderChange

Error Message:
Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>

Stack Trace:
org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>
	at __randomizedtesting.SeedInfo.seed([72F707B83BC34278:ACA4804F215BB789]:0)
	at org.junit.Assert.assertEquals(Assert.java:125)
	at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:493)
	at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:652)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 8650 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4:junit4]   2> 0 T982 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351372596646
[junit4:junit4]   2> 0 T982 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 3 T982 oas.SolrTestCaseJ4.setUp ###Starting testPlaceholders
[junit4:junit4]   2> 3 T982 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 4 T983 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 4 T983 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 5 T983 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 104 T982 oasc.ZkTestServer.run start zk server on port:60434
[junit4:junit4]   2> 104 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:60434/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1db18d0
[junit4:junit4]   2> 104 T988 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:60434
[junit4:junit4]   2> 104 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 105 T988 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:60434, initiating session
[junit4:junit4]   2> 105 T984 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:43716
[junit4:junit4]   2> 105 T984 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:43716
[junit4:junit4]   2> 106 T986 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 137 T986 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41565ab0000 with negotiated timeout 10000 for client /127.0.0.1:43716
[junit4:junit4]   2> 137 T988 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:60434, sessionid = 0x13aa41565ab0000, negotiated timeout = 10000
[junit4:junit4]   2> 137 T989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1db18d0 name:ZooKeeperConnection Watcher:127.0.0.1:60434/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 137 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 138 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:60434 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1206a8a
[junit4:junit4]   2> 138 T990 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:60434
[junit4:junit4]   2> 138 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 139 T990 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:60434, initiating session
[junit4:junit4]   2> 139 T984 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:43717
[junit4:junit4]   2> 139 T984 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:43717
[junit4:junit4]   2> 141 T986 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41565ab0001 with negotiated timeout 10000 for client /127.0.0.1:43717
[junit4:junit4]   2> 141 T990 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:60434, sessionid = 0x13aa41565ab0001, negotiated timeout = 10000
[junit4:junit4]   2> 141 T991 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1206a8a name:ZooKeeperConnection Watcher:127.0.0.1:60434 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 141 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 142 T987 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41565ab0001
[junit4:junit4]   2> 142 T991 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 143 T984 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43717 which had sessionid 0x13aa41565ab0001
[junit4:junit4]   2> 142 T982 oaz.ZooKeeper.close Session: 0x13aa41565ab0001 closed
[junit4:junit4]   2> 143 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:60434 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@cb865b
[junit4:junit4]   2> 144 T992 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:60434
[junit4:junit4]   2> 144 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 144 T992 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:60434, initiating session
[junit4:junit4]   2> 144 T984 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:43718
[junit4:junit4]   2> 145 T984 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:43718
[junit4:junit4]   2> 145 T986 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41565ab0002 with negotiated timeout 10000 for client /127.0.0.1:43718
[junit4:junit4]   2> 145 T992 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:60434, sessionid = 0x13aa41565ab0002, negotiated timeout = 10000
[junit4:junit4]   2> 146 T993 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cb865b name:ZooKeeperConnection Watcher:127.0.0.1:60434 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 146 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 146 T982 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 148 T987 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41565ab0002
[junit4:junit4]   2> 148 T984 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43718 which had sessionid 0x13aa41565ab0002
[junit4:junit4]   2> 148 T982 oaz.ZooKeeper.close Session: 0x13aa41565ab0002 closed
[junit4:junit4]   2> 149 T982 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 149 T993 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 150 T982 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 151 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 153 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:60434/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@18f3384
[junit4:junit4]   2> 153 T994 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:60434
[junit4:junit4]   2> 153 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 154 T994 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:60434, initiating session
[junit4:junit4]   2> 154 T984 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:43719
[junit4:junit4]   2> 154 T984 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:43719
[junit4:junit4]   2> 155 T986 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41565ab0003 with negotiated timeout 10000 for client /127.0.0.1:43719
[junit4:junit4]   2> 155 T994 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:60434, sessionid = 0x13aa41565ab0003, negotiated timeout = 10000
[junit4:junit4]   2> 155 T995 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18f3384 name:ZooKeeperConnection Watcher:127.0.0.1:60434/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 155 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 156 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 157 T982 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 159 T995 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 159 T989 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 159 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:60434/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@10c1409
[junit4:junit4]   2> 160 T996 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:60434
[junit4:junit4]   2> 160 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 161 T996 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:60434, initiating session
[junit4:junit4]   2> 161 T984 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:43720
[junit4:junit4]   2> 161 T984 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:43720
[junit4:junit4]   2> 162 T986 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41565ab0004 with negotiated timeout 10000 for client /127.0.0.1:43720
[junit4:junit4]   2> 162 T996 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:60434, sessionid = 0x13aa41565ab0004, negotiated timeout = 10000
[junit4:junit4]   2> 162 T997 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10c1409 name:ZooKeeperConnection Watcher:127.0.0.1:60434/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 162 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 163 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 166 T987 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41565ab0004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 166 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 167 T982 oasc.Overseer.start Overseer (id=88563554494119940-127.0.0.1:60434_solr-n_0000000000) starting
[junit4:junit4]   2> 168 T987 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41565ab0004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 169 T987 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41565ab0004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 169 T987 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41565ab0004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 170 T999 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 170 T987 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41565ab0003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 171 T998 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 171 T987 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41565ab0003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 675 T998 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 675 T998 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=12 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"12",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 676 T998 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 12
[junit4:junit4]   2> 676 T987 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41565ab0004 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 680 T995 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 680 T989 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 1173 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 1180 T987 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41565ab0003 type:delete cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 1180 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 1183 T987 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41565ab0003 type:create cxid:0x24 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1184 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1185 T987 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41565ab0004
[junit4:junit4]   2> 1185 T997 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1185 T982 oaz.ZooKeeper.close Session: 0x13aa41565ab0004 closed
[junit4:junit4]   2> 1185 T984 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43720 which had sessionid 0x13aa41565ab0004
[junit4:junit4]   2> 1187 T987 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41565ab0003
[junit4:junit4]   2> 1187 T995 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 1187 T989 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 1187 T995 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1187 T982 oaz.ZooKeeper.close Session: 0x13aa41565ab0003 closed
[junit4:junit4]   2> 1187 T984 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43719 which had sessionid 0x13aa41565ab0003
[junit4:junit4]   2> 1188 T987 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41565ab0000
[junit4:junit4]   2> 1188 T989 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1188 T984 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:43716 which had sessionid 0x13aa41565ab0000
[junit4:junit4]   2> 1188 T982 oaz.ZooKeeper.close Session: 0x13aa41565ab0000 closed
[junit4:junit4]   2> 1188 T986 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 1188 T987 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 1189 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 1190 T984 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 1191 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 1191 T982 oas.SolrTestCaseJ4.tearDown ###Ending testPlaceholders
[junit4:junit4]   2> 1194 T982 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4:junit4]   2> 1195 T982 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 1195 T1000 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 1195 T1000 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 1196 T1000 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351372596646/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 1197 T1000 oazsp.FileTxnSnapLog.save Snapshotting: 2e
[junit4:junit4]   2> 1295 T982 oasc.ZkTestServer.run start zk server on port:56758
[junit4:junit4]   2> 1296 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56758 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1276c95
[junit4:junit4]   2> 1296 T1005 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56758
[junit4:junit4]   2> 1296 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1297 T1005 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56758, initiating session
[junit4:junit4]   2> 1297 T1001 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40848
[junit4:junit4]   2> 1297 T1001 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40848
[junit4:junit4]   2> 1297 T1003 oazsp.FileTxnLog.append Creating new log file: log.2f
[junit4:junit4]   2> 1298 T1003 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4156a540000 with negotiated timeout 10000 for client /127.0.0.1:40848
[junit4:junit4]   2> 1299 T1005 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56758, sessionid = 0x13aa4156a540000, negotiated timeout = 10000
[junit4:junit4]   2> 1299 T1006 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1276c95 name:ZooKeeperConnection Watcher:127.0.0.1:56758 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1299 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1309 T1004 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4156a540000
[junit4:junit4]   2> 1309 T1006 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1310 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40848 which had sessionid 0x13aa4156a540000
[junit4:junit4]   2> 1309 T982 oaz.ZooKeeper.close Session: 0x13aa4156a540000 closed
[junit4:junit4]   2> 1310 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56758 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@7a6435
[junit4:junit4]   2> 1310 T1007 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56758
[junit4:junit4]   2> 1310 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1311 T1007 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56758, initiating session
[junit4:junit4]   2> 1311 T1001 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40849
[junit4:junit4]   2> 1311 T1001 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40849
[junit4:junit4]   2> 1312 T1003 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4156a540001 with negotiated timeout 10000 for client /127.0.0.1:40849
[junit4:junit4]   2> 1312 T1007 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56758, sessionid = 0x13aa4156a540001, negotiated timeout = 10000
[junit4:junit4]   2> 1312 T1008 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a6435 name:ZooKeeperConnection Watcher:127.0.0.1:56758 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1312 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1312 T982 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 1314 T1004 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4156a540001
[junit4:junit4]   2> 1314 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40849 which had sessionid 0x13aa4156a540001
[junit4:junit4]   2> 1314 T982 oaz.ZooKeeper.close Session: 0x13aa4156a540001 closed
[junit4:junit4]   2> 1315 T1008 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1315 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56758/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1f47ab7
[junit4:junit4]   2> 1315 T1009 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56758
[junit4:junit4]   2> 1315 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1316 T1009 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56758, initiating session
[junit4:junit4]   2> 1316 T1001 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40850
[junit4:junit4]   2> 1316 T1001 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40850
[junit4:junit4]   2> 1317 T1003 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4156a540002 with negotiated timeout 10000 for client /127.0.0.1:40850
[junit4:junit4]   2> 1317 T1009 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56758, sessionid = 0x13aa4156a540002, negotiated timeout = 10000
[junit4:junit4]   2> 1317 T1010 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f47ab7 name:ZooKeeperConnection Watcher:127.0.0.1:56758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1317 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1317 T982 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1318 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56758/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@7fb846
[junit4:junit4]   2> 1319 T1011 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56758
[junit4:junit4]   2> 1319 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1319 T1011 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56758, initiating session
[junit4:junit4]   2> 1319 T1001 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40851
[junit4:junit4]   2> 1320 T1001 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40851
[junit4:junit4]   2> 1320 T1003 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4156a540003 with negotiated timeout 10000 for client /127.0.0.1:40851
[junit4:junit4]   2> 1320 T1011 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56758, sessionid = 0x13aa4156a540003, negotiated timeout = 10000
[junit4:junit4]   2> 1321 T1012 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7fb846 name:ZooKeeperConnection Watcher:127.0.0.1:56758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1321 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1321 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1324 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1324 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1325 T982 oasc.Overseer.start Overseer (id=88563554572304387-127.0.0.1:56758_solr-n_0000000000) starting
[junit4:junit4]   2> 1326 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1327 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1327 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1329 T1014 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 1330 T1013 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1330 T982 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1331 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1332 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56758/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@7dbf04
[junit4:junit4]   2> 1333 T1015 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56758
[junit4:junit4]   2> 1333 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1333 T1015 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56758, initiating session
[junit4:junit4]   2> 1333 T1001 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40852
[junit4:junit4]   2> 1334 T1001 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40852
[junit4:junit4]   2> 1334 T1003 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4156a540004 with negotiated timeout 10000 for client /127.0.0.1:40852
[junit4:junit4]   2> 1334 T1015 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56758, sessionid = 0x13aa4156a540004, negotiated timeout = 10000
[junit4:junit4]   2> 1335 T1016 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7dbf04 name:ZooKeeperConnection Watcher:127.0.0.1:56758/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1335 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1335 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1336 T982 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4:junit4]   2> 1339 T1016 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 1339 T1010 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 1339 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1340 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 1354 T985 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 1683 T998 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 1683 T998 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88563554494119940-127.0.0.1:60434_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 1836 T1013 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1836 T1013 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
[junit4:junit4]   2> 1837 T1013 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4]   2> 1837 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540003 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 1841 T1016 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 1841 T1010 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2341 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 2350 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:delete cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 2350 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 2353 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x24 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2354 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x26 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2848 T1013 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2850 T1013 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core2",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
[junit4:junit4]   2> 2852 T1016 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2852 T1010 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 2855 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 2860 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:delete cxid:0x33 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 2860 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 2862 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x3a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2863 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x3c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3356 T1013 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3359 T1013 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core3",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
[junit4:junit4]   2> 3361 T1016 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3361 T1010 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3364 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 3369 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:delete cxid:0x49 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard3 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard3
[junit4:junit4]   2> 3369 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4]   2> 3371 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x50 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3372 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x52 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3864 T1013 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3867 T1013 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core4",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
[junit4:junit4]   2> 3870 T1010 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3870 T1016 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3873 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 3876 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x5a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 3877 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x5e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4374 T1013 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4374 T1013 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core5",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
[junit4:junit4]   2> 4379 T1016 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 4379 T1010 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 4878 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 4881 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x66 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard2/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 4882 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x6a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5384 T1013 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5385 T1013 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core6",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr/"}
[junit4:junit4]   2> 5388 T1016 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 5388 T1010 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 5884 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 5886 T1004 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4156a540004 type:create cxid:0x72 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard3/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 5888 T1004 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4156a540002
[junit4:junit4]   2> 5888 T1010 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 5889 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40850 which had sessionid 0x13aa4156a540002
[junit4:junit4]   2> 5888 T982 oaz.ZooKeeper.close Session: 0x13aa4156a540002 closed
[junit4:junit4]   2> 5890 T1004 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4156a540004
[junit4:junit4]   2> 5890 T1016 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 5890 T982 oaz.ZooKeeper.close Session: 0x13aa4156a540004 closed
[junit4:junit4]   2> 5891 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40852 which had sessionid 0x13aa4156a540004
[junit4:junit4]   2> 5891 T1004 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4156a540003
[junit4:junit4]   2> 5892 T1012 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 5892 T1001 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40851 which had sessionid 0x13aa4156a540003
[junit4:junit4]   2> 5892 T982 oaz.ZooKeeper.close Session: 0x13aa4156a540003 closed
[junit4:junit4]   2> 5892 T1004 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 5892 T1003 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 5893 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 5894 T1001 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 5894 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 5894 T982 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4:junit4]   2> 5898 T982 oas.SolrTestCaseJ4.setUp ###Starting testStateChange
[junit4:junit4]   2> 5899 T982 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 5899 T1017 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 5899 T1017 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 5900 T1017 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351372596646/zookeeper/server1/data/version-2/snapshot.2e
[junit4:junit4]   2> 5904 T1017 oazsp.FileTxnSnapLog.save Snapshotting: a7
[junit4:junit4]   2> 5999 T982 oasc.ZkTestServer.run start zk server on port:33817
[junit4:junit4]   2> 6000 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:33817/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@de8b19
[junit4:junit4]   2> 6000 T1022 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:33817
[junit4:junit4]   2> 6000 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6001 T1022 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:33817, initiating session
[junit4:junit4]   2> 6001 T1018 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42037
[junit4:junit4]   2> 6001 T1018 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42037
[junit4:junit4]   2> 6001 T1020 oazsp.FileTxnLog.append Creating new log file: log.a8
[junit4:junit4]   2> 6003 T1020 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4157cb70000 with negotiated timeout 10000 for client /127.0.0.1:42037
[junit4:junit4]   2> 6003 T1022 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:33817, sessionid = 0x13aa4157cb70000, negotiated timeout = 10000
[junit4:junit4]   2> 6003 T1023 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@de8b19 name:ZooKeeperConnection Watcher:127.0.0.1:33817/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6004 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6004 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:33817 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@40a241
[junit4:junit4]   2> 6005 T1024 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:33817
[junit4:junit4]   2> 6005 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6005 T1024 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:33817, initiating session
[junit4:junit4]   2> 6005 T1018 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42038
[junit4:junit4]   2> 6005 T1018 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42038
[junit4:junit4]   2> 6006 T1020 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4157cb70001 with negotiated timeout 10000 for client /127.0.0.1:42038
[junit4:junit4]   2> 6006 T1024 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:33817, sessionid = 0x13aa4157cb70001, negotiated timeout = 10000
[junit4:junit4]   2> 6006 T1025 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40a241 name:ZooKeeperConnection Watcher:127.0.0.1:33817 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6006 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6021 T1021 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4157cb70001
[junit4:junit4]   2> 6021 T1018 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:42038 which had sessionid 0x13aa4157cb70001
[junit4:junit4]   2> 6021 T982 oaz.ZooKeeper.close Session: 0x13aa4157cb70001 closed
[junit4:junit4]   2> 6022 T1025 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 6022 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:33817 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@d4ecd5
[junit4:junit4]   2> 6022 T1026 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:33817
[junit4:junit4]   2> 6022 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6023 T1026 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:33817, initiating session
[junit4:junit4]   2> 6023 T1018 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42039
[junit4:junit4]   2> 6023 T1018 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42039
[junit4:junit4]   2> 6024 T1020 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4157cb70002 with negotiated timeout 10000 for client /127.0.0.1:42039
[junit4:junit4]   2> 6024 T1026 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:33817, sessionid = 0x13aa4157cb70002, negotiated timeout = 10000
[junit4:junit4]   2> 6024 T1027 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d4ecd5 name:ZooKeeperConnection Watcher:127.0.0.1:33817 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6024 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6024 T982 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 6026 T1021 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4157cb70002
[junit4:junit4]   2> 6026 T1018 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:42039 which had sessionid 0x13aa4157cb70002
[junit4:junit4]   2> 6026 T982 oaz.ZooKeeper.close Session: 0x13aa4157cb70002 closed
[junit4:junit4]   2> 6026 T1027 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 6026 T982 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 6028 T982 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 6029 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 6030 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:33817/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@336d4
[junit4:junit4]   2> 6030 T1028 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:33817
[junit4:junit4]   2> 6030 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6031 T1028 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:33817, initiating session
[junit4:junit4]   2> 6031 T1018 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:42040
[junit4:junit4]   2> 6031 T1018 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:42040
[junit4:junit4]   2> 6032 T1020 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4157cb70003 with negotiated timeout 10000 for client /127.0.0.1:42040
[junit4:junit4]   2> 6032 T1028 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:33817, sessionid = 0x13aa4157cb70003, negotiated timeout = 10000
[junit4:junit4]   2> 6032 T1029 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@336d4 name:ZooKeeperConnection Watcher:127.0.0.1:33817/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6032 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6033 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 6035 T1021 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4157cb70003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 6035 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 6037 T982 oasc.Overseer.start Overseer (id=88563554880782339-127.0.0.1:33817_solr-n_0000000000) starting
[junit4:junit4]   2> 6037 T1021 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4157cb70003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6038 T1021 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4157cb70003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6038 T1021 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4157cb70003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6039 T1031 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 6039 T1021 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4157cb70000 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6040 T1021 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4157cb70000 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 6040 T1030 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 6042 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6093 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6144 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6196 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6248 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6299 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6350 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6354 T1002 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 6391 T1013 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 6392 T1013 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88563554572304387-127.0.0.1:56758_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 6402 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6453 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6504 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6543 T1030 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6544 T1030 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "roles":""}
[junit4:junit4]   2> 6544 T1021 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4157cb70003 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 6547 T1023 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 6555 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6891 T1016 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 6891 T1016 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 6892 T1016 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 6892 T1016 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 7050 T1030 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7051 T1030 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "roles":""}
[junit4:junit4]   2> 7053 T1023 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 7060 T1021 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4157cb70000
[junit4:junit4]   2> 7060 T1023 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 7061 T1018 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:42037 which had sessionid 0x13aa4157cb70000
[junit4:junit4]   2> 7060 T982 oaz.ZooKeeper.close Session: 0x13aa4157cb70000 closed
[junit4:junit4]   2> 7061 T1021 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4157cb70003
[junit4:junit4]   2> 7062 T1029 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 7062 T1018 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:42040 which had sessionid 0x13aa4157cb70003
[junit4:junit4]   2> 7062 T982 oaz.ZooKeeper.close Session: 0x13aa4157cb70003 closed
[junit4:junit4]   2> 7062 T1021 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 7062 T1020 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 7063 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 7064 T1018 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 7065 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 7065 T982 oas.SolrTestCaseJ4.tearDown ###Ending testStateChange
[junit4:junit4]   2> 7069 T982 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure
[junit4:junit4]   2> 7069 T982 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 7069 T1032 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 7069 T1032 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 7070 T1032 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351372596646/zookeeper/server1/data/version-2/snapshot.a7
[junit4:junit4]   2> 7074 T1032 oazsp.FileTxnSnapLog.save Snapshotting: de
[junit4:junit4]   2> 7169 T982 oasc.ZkTestServer.run start zk server on port:42631
[junit4:junit4]   2> 7170 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42631/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@228796
[junit4:junit4]   2> 7170 T1037 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42631
[junit4:junit4]   2> 7170 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7171 T1037 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42631, initiating session
[junit4:junit4]   2> 7171 T1033 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59983
[junit4:junit4]   2> 7171 T1033 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59983
[junit4:junit4]   2> 7171 T1035 oazsp.FileTxnLog.append Creating new log file: log.df
[junit4:junit4]   2> 7196 T1035 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41581480000 with negotiated timeout 10000 for client /127.0.0.1:59983
[junit4:junit4]   2> 7196 T1037 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42631, sessionid = 0x13aa41581480000, negotiated timeout = 10000
[junit4:junit4]   2> 7197 T1038 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@228796 name:ZooKeeperConnection Watcher:127.0.0.1:42631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7197 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7197 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42631 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@69bc3b
[junit4:junit4]   2> 7198 T1039 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42631
[junit4:junit4]   2> 7198 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7198 T1039 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42631, initiating session
[junit4:junit4]   2> 7198 T1033 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59984
[junit4:junit4]   2> 7198 T1033 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59984
[junit4:junit4]   2> 7200 T1035 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41581480001 with negotiated timeout 10000 for client /127.0.0.1:59984
[junit4:junit4]   2> 7200 T1039 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42631, sessionid = 0x13aa41581480001, negotiated timeout = 10000
[junit4:junit4]   2> 7200 T1040 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69bc3b name:ZooKeeperConnection Watcher:127.0.0.1:42631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7201 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7207 T1036 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41581480001
[junit4:junit4]   2> 7208 T1040 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 7208 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59984 which had sessionid 0x13aa41581480001
[junit4:junit4]   2> 7208 T982 oaz.ZooKeeper.close Session: 0x13aa41581480001 closed
[junit4:junit4]   2> 7208 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42631 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@fbe962
[junit4:junit4]   2> 7209 T1041 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42631
[junit4:junit4]   2> 7209 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7209 T1041 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42631, initiating session
[junit4:junit4]   2> 7209 T1033 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59985
[junit4:junit4]   2> 7209 T1033 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59985
[junit4:junit4]   2> 7210 T1035 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41581480002 with negotiated timeout 10000 for client /127.0.0.1:59985
[junit4:junit4]   2> 7210 T1041 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42631, sessionid = 0x13aa41581480002, negotiated timeout = 10000
[junit4:junit4]   2> 7210 T1042 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fbe962 name:ZooKeeperConnection Watcher:127.0.0.1:42631 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7210 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7210 T982 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 7212 T1036 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41581480002
[junit4:junit4]   2> 7212 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59985 which had sessionid 0x13aa41581480002
[junit4:junit4]   2> 7212 T982 oaz.ZooKeeper.close Session: 0x13aa41581480002 closed
[junit4:junit4]   2> 7212 T1042 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 7213 T982 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 7214 T982 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 7215 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 7216 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42631/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@77420f
[junit4:junit4]   2> 7216 T1043 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42631
[junit4:junit4]   2> 7216 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7217 T1043 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42631, initiating session
[junit4:junit4]   2> 7217 T1033 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59986
[junit4:junit4]   2> 7217 T1033 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59986
[junit4:junit4]   2> 7218 T1035 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41581480003 with negotiated timeout 10000 for client /127.0.0.1:59986
[junit4:junit4]   2> 7218 T1043 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42631, sessionid = 0x13aa41581480003, negotiated timeout = 10000
[junit4:junit4]   2> 7218 T1044 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77420f name:ZooKeeperConnection Watcher:127.0.0.1:42631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7218 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7219 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 7219 T982 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 7221 T1044 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 7221 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42631/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@3925c
[junit4:junit4]   2> 7221 T1038 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 7221 T1045 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42631
[junit4:junit4]   2> 7221 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7222 T1045 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42631, initiating session
[junit4:junit4]   2> 7222 T1033 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59987
[junit4:junit4]   2> 7222 T1033 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59987
[junit4:junit4]   2> 7223 T1035 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41581480004 with negotiated timeout 10000 for client /127.0.0.1:59987
[junit4:junit4]   2> 7223 T1045 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42631, sessionid = 0x13aa41581480004, negotiated timeout = 10000
[junit4:junit4]   2> 7223 T1046 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3925c name:ZooKeeperConnection Watcher:127.0.0.1:42631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7223 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7224 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 7226 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 7226 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 7228 T982 oasc.Overseer.start Overseer (id=88563554957393924-127.0.0.1:42631_solr-n_0000000000) starting
[junit4:junit4]   2> 7228 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7229 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7229 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7230 T1048 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 7231 T1047 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 7354 T1019 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 7554 T1030 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 7555 T1030 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88563554880782339-127.0.0.1:33817_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 8230 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8247 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 8758 T1047 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8759 T1047 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 8759 T1047 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 8760 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480004 type:create cxid:0x26 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 8764 T1044 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 8764 T1038 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 9258 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 9791 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:delete cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 9792 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 9794 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0x24 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 9796 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9798 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0x26 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 9799 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 9802 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0x2d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 10290 T1047 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10292 T1047 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 10295 T1038 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 10295 T1044 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 10346 T1036 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41581480004
[junit4:junit4]   2> 10347 T1046 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 10347 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59987 which had sessionid 0x13aa41581480004
[junit4:junit4]   2> 10347 T982 oaz.ZooKeeper.close Session: 0x13aa41581480004 closed
[junit4:junit4]   2> 10797 T1047 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 10798 T1047 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88563554957393924-127.0.0.1:42631_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 11348 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0x32 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11350 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 11351 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0x39 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 11359 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:42631/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@655d78
[junit4:junit4]   2> 11360 T1049 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:42631
[junit4:junit4]   2> 11360 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 11360 T1049 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:42631, initiating session
[junit4:junit4]   2> 11360 T1033 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59994
[junit4:junit4]   2> 11361 T1033 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59994
[junit4:junit4]   2> 11361 T1035 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa41581480005 with negotiated timeout 10000 for client /127.0.0.1:59994
[junit4:junit4]   2> 11361 T1049 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:42631, sessionid = 0x13aa41581480005, negotiated timeout = 10000
[junit4:junit4]   2> 11362 T1050 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@655d78 name:ZooKeeperConnection Watcher:127.0.0.1:42631/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 11362 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 11364 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 11364 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 11367 T982 oasc.Overseer.start Overseer (id=88563554957393925-127.0.0.1:42631_solr-n_0000000002) starting
[junit4:junit4]   2> 11368 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480005 type:create cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11369 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11369 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480005 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11370 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480005 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11378 T1052 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 11379 T1051 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 11380 T1051 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11381 T1051 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 11384 T1038 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 11384 T1044 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 11435 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0x3e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11447 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 11449 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0x45 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 11888 T1051 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11892 T1044 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 11895 T1036 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa41581480003 type:create cxid:0x51 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11896 T1044 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 11897 T1044 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 11901 T1051 oasc.Overseer$ClusterStateUpdater.setShardLeader SEVERE Could not mark shard leader for non existing collection:collection1
[junit4:junit4]   2> 11903 T1038 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 11903 T1044 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 12405 T1036 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41581480003
[junit4:junit4]   2> 12405 T1044 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 12405 T1038 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 12405 T1044 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12405 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59986 which had sessionid 0x13aa41581480003
[junit4:junit4]   2> 12405 T982 oaz.ZooKeeper.close Session: 0x13aa41581480003 closed
[junit4:junit4]   2> 12406 T1036 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41581480005
[junit4:junit4]   2> 12407 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59994 which had sessionid 0x13aa41581480005
[junit4:junit4]   2> 12407 T982 oaz.ZooKeeper.close Session: 0x13aa41581480005 closed
[junit4:junit4]   2> 12407 T1050 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12408 T1036 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa41581480000
[junit4:junit4]   2> 12408 T1038 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12408 T1033 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59983 which had sessionid 0x13aa41581480000
[junit4:junit4]   2> 12408 T982 oaz.ZooKeeper.close Session: 0x13aa41581480000 closed
[junit4:junit4]   2> 12409 T1035 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 12409 T1036 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 12409 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 12410 T1033 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 12411 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 12411 T982 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4:junit4]   2> 12415 T982 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
[junit4:junit4]   2> 12415 T982 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 12415 T1053 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 12416 T1053 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 12416 T1053 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351372596646/zookeeper/server1/data/version-2/snapshot.de
[junit4:junit4]   2> 12420 T1053 oazsp.FileTxnSnapLog.save Snapshotting: 14a
[junit4:junit4]   2> 12488 T1051 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 12489 T1051 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88563554957393925-127.0.0.1:42631_solr-n_0000000002) am no longer a leader.
[junit4:junit4]   2> 12515 T982 oasc.ZkTestServer.run start zk server on port:52368
[junit4:junit4]   2> 12516 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:52368/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@358f9a
[junit4:junit4]   2> 12516 T1058 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:52368
[junit4:junit4]   2> 12516 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12517 T1058 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:52368, initiating session
[junit4:junit4]   2> 12517 T1054 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53971
[junit4:junit4]   2> 12517 T1054 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53971
[junit4:junit4]   2> 12518 T1056 oazsp.FileTxnLog.append Creating new log file: log.14b
[junit4:junit4]   2> 12587 T1056 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa415962a0000 with negotiated timeout 10000 for client /127.0.0.1:53971
[junit4:junit4]   2> 12587 T1058 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:52368, sessionid = 0x13aa415962a0000, negotiated timeout = 10000
[junit4:junit4]   2> 12587 T1059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@358f9a name:ZooKeeperConnection Watcher:127.0.0.1:52368/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12587 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12588 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:52368 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@19e5bfc
[junit4:junit4]   2> 12588 T1060 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:52368
[junit4:junit4]   2> 12588 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12589 T1060 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:52368, initiating session
[junit4:junit4]   2> 12589 T1054 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53972
[junit4:junit4]   2> 12589 T1054 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53972
[junit4:junit4]   2> 12591 T1056 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa415962a0001 with negotiated timeout 10000 for client /127.0.0.1:53972
[junit4:junit4]   2> 12591 T1060 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:52368, sessionid = 0x13aa415962a0001, negotiated timeout = 10000
[junit4:junit4]   2> 12591 T1061 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19e5bfc name:ZooKeeperConnection Watcher:127.0.0.1:52368 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12591 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12597 T1057 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa415962a0001
[junit4:junit4]   2> 12598 T1054 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53972 which had sessionid 0x13aa415962a0001
[junit4:junit4]   2> 12598 T982 oaz.ZooKeeper.close Session: 0x13aa415962a0001 closed
[junit4:junit4]   2> 12598 T1061 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12598 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:52368 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@ced6cd
[junit4:junit4]   2> 12599 T1062 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:52368
[junit4:junit4]   2> 12599 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12599 T1062 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:52368, initiating session
[junit4:junit4]   2> 12599 T1054 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53973
[junit4:junit4]   2> 12600 T1054 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53973
[junit4:junit4]   2> 12600 T1056 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa415962a0002 with negotiated timeout 10000 for client /127.0.0.1:53973
[junit4:junit4]   2> 12600 T1062 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:52368, sessionid = 0x13aa415962a0002, negotiated timeout = 10000
[junit4:junit4]   2> 12601 T1063 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ced6cd name:ZooKeeperConnection Watcher:127.0.0.1:52368 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12601 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12601 T982 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 12602 T1057 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa415962a0002
[junit4:junit4]   2> 12603 T1054 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53973 which had sessionid 0x13aa415962a0002
[junit4:junit4]   2> 12603 T982 oaz.ZooKeeper.close Session: 0x13aa415962a0002 closed
[junit4:junit4]   2> 12603 T1063 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12603 T982 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 12604 T982 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 12605 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 12606 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:52368/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1d1d08e
[junit4:junit4]   2> 12607 T1064 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:52368
[junit4:junit4]   2> 12607 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12607 T1064 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:52368, initiating session
[junit4:junit4]   2> 12607 T1054 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53974
[junit4:junit4]   2> 12607 T1054 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53974
[junit4:junit4]   2> 12608 T1056 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa415962a0003 with negotiated timeout 10000 for client /127.0.0.1:53974
[junit4:junit4]   2> 12608 T1064 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:52368, sessionid = 0x13aa415962a0003, negotiated timeout = 10000
[junit4:junit4]   2> 12608 T1065 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d1d08e name:ZooKeeperConnection Watcher:127.0.0.1:52368/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12608 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12609 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 12610 T982 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 12611 T1065 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 12611 T1059 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 12611 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:52368/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@15fe85c
[junit4:junit4]   2> 12612 T1066 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:52368
[junit4:junit4]   2> 12612 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12612 T1066 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:52368, initiating session
[junit4:junit4]   2> 12612 T1054 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53975
[junit4:junit4]   2> 12612 T1054 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53975
[junit4:junit4]   2> 12613 T1056 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa415962a0004 with negotiated timeout 10000 for client /127.0.0.1:53975
[junit4:junit4]   2> 12613 T1066 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:52368, sessionid = 0x13aa415962a0004, negotiated timeout = 10000
[junit4:junit4]   2> 12613 T1067 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15fe85c name:ZooKeeperConnection Watcher:127.0.0.1:52368/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12613 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12614 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 12616 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 12616 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 12618 T982 oasc.Overseer.start Overseer (id=88563555307749380-127.0.0.1:52368_solr-n_0000000000) starting
[junit4:junit4]   2> 12618 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12619 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12619 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12620 T1069 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 12620 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0003 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12621 T1068 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 12621 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0003 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 13123 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13124 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 13124 T1068 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 13125 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0004 type:create cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 13128 T1059 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 13128 T1065 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 13354 T1034 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 13622 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 13630 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0003 type:delete cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 13631 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 13633 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0003 type:create cxid:0x24 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13634 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13635 T1057 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa415962a0003
[junit4:junit4]   2> 13635 T1065 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 13637 T1065 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 13637 T1059 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 13637 T1054 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53974 which had sessionid 0x13aa415962a0003
[junit4:junit4]   2> 13637 T982 oaz.ZooKeeper.close Session: 0x13aa415962a0003 closed
[junit4:junit4]   2> 13638 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:52368/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@9a6ae3
[junit4:junit4]   2> 13638 T1070 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:52368
[junit4:junit4]   2> 13638 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13639 T1070 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:52368, initiating session
[junit4:junit4]   2> 13639 T1054 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:53978
[junit4:junit4]   2> 13639 T1054 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:53978
[junit4:junit4]   2> 13641 T1056 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa415962a0005 with negotiated timeout 10000 for client /127.0.0.1:53978
[junit4:junit4]   2> 13641 T1070 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:52368, sessionid = 0x13aa415962a0005, negotiated timeout = 10000
[junit4:junit4]   2> 13641 T1071 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9a6ae3 name:ZooKeeperConnection Watcher:127.0.0.1:52368/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13641 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13642 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 13643 T982 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 13645 T1059 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 13645 T1071 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 13645 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13646 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 13647 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0005 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 13648 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0005 type:delete cxid:0x14 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard1 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard1
[junit4:junit4]   2> 13649 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 13650 T1057 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa415962a0005 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14131 T1068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14134 T1068 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 14138 T1059 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 14138 T1071 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 14139 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14140 T1057 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa415962a0004
[junit4:junit4]   2> 14141 T982 oaz.ZooKeeper.close Session: 0x13aa415962a0004 closed
[junit4:junit4]   2> 14141 T1054 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53975 which had sessionid 0x13aa415962a0004
[junit4:junit4]   2> 14141 T1067 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14142 T1057 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa415962a0005
[junit4:junit4]   2> 14142 T1071 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 14142 T1059 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 14142 T1071 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14142 T1054 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53978 which had sessionid 0x13aa415962a0005
[junit4:junit4]   2> 14142 T982 oaz.ZooKeeper.close Session: 0x13aa415962a0005 closed
[junit4:junit4]   2> 14143 T1057 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa415962a0000
[junit4:junit4]   2> 14143 T1059 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14143 T982 oaz.ZooKeeper.close Session: 0x13aa415962a0000 closed
[junit4:junit4]   2> 14143 T1054 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:53971 which had sessionid 0x13aa415962a0000
[junit4:junit4]   2> 14144 T1056 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 14144 T1057 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 14144 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 14146 T1054 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 14146 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 14147 T982 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
[junit4:junit4]   2> 14150 T982 oas.SolrTestCaseJ4.setUp ###Starting testReplay
[junit4:junit4]   2> 14151 T982 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 14151 T1072 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 14151 T1072 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 14152 T1072 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351372596646/zookeeper/server1/data/version-2/snapshot.14a
[junit4:junit4]   2> 14155 T1072 oazsp.FileTxnSnapLog.save Snapshotting: 196
[junit4:junit4]   2> 14251 T982 oasc.ZkTestServer.run start zk server on port:58248
[junit4:junit4]   2> 14252 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58248/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1f52aaf
[junit4:junit4]   2> 14252 T1077 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:58248
[junit4:junit4]   2> 14252 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14253 T1077 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58248, initiating session
[junit4:junit4]   2> 14253 T1073 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59362
[junit4:junit4]   2> 14254 T1073 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59362
[junit4:junit4]   2> 14254 T1075 oazsp.FileTxnLog.append Creating new log file: log.197
[junit4:junit4]   2> 14256 T1075 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159cf20000 with negotiated timeout 10000 for client /127.0.0.1:59362
[junit4:junit4]   2> 14256 T1077 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:58248, sessionid = 0x13aa4159cf20000, negotiated timeout = 10000
[junit4:junit4]   2> 14256 T1078 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f52aaf name:ZooKeeperConnection Watcher:127.0.0.1:58248/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14256 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14257 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58248 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1ed6274
[junit4:junit4]   2> 14257 T1079 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:58248
[junit4:junit4]   2> 14257 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14258 T1079 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58248, initiating session
[junit4:junit4]   2> 14258 T1073 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59363
[junit4:junit4]   2> 14258 T1073 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59363
[junit4:junit4]   2> 14259 T1075 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159cf20001 with negotiated timeout 10000 for client /127.0.0.1:59363
[junit4:junit4]   2> 14260 T1079 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:58248, sessionid = 0x13aa4159cf20001, negotiated timeout = 10000
[junit4:junit4]   2> 14260 T1080 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ed6274 name:ZooKeeperConnection Watcher:127.0.0.1:58248 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14260 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14273 T1076 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159cf20001
[junit4:junit4]   2> 14273 T1073 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59363 which had sessionid 0x13aa4159cf20001
[junit4:junit4]   2> 14274 T1080 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14274 T982 oaz.ZooKeeper.close Session: 0x13aa4159cf20001 closed
[junit4:junit4]   2> 14274 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58248 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@19d0d16
[junit4:junit4]   2> 14275 T1081 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:58248
[junit4:junit4]   2> 14275 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14276 T1081 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58248, initiating session
[junit4:junit4]   2> 14276 T1073 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59364
[junit4:junit4]   2> 14276 T1073 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59364
[junit4:junit4]   2> 14277 T1075 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159cf20002 with negotiated timeout 10000 for client /127.0.0.1:59364
[junit4:junit4]   2> 14277 T1081 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:58248, sessionid = 0x13aa4159cf20002, negotiated timeout = 10000
[junit4:junit4]   2> 14277 T1082 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19d0d16 name:ZooKeeperConnection Watcher:127.0.0.1:58248 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14278 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14278 T982 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 14279 T1076 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159cf20002
[junit4:junit4]   2> 14280 T1073 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59364 which had sessionid 0x13aa4159cf20002
[junit4:junit4]   2> 14280 T982 oaz.ZooKeeper.close Session: 0x13aa4159cf20002 closed
[junit4:junit4]   2> 14281 T982 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 14281 T1082 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14282 T982 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 14283 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 14285 T1076 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159cf20000 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 14287 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:58248/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@355f70
[junit4:junit4]   2> 14288 T1083 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:58248
[junit4:junit4]   2> 14288 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14288 T1083 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:58248, initiating session
[junit4:junit4]   2> 14288 T1073 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:59365
[junit4:junit4]   2> 14289 T1073 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:59365
[junit4:junit4]   2> 14290 T1075 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159cf20003 with negotiated timeout 10000 for client /127.0.0.1:59365
[junit4:junit4]   2> 14290 T1083 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:58248, sessionid = 0x13aa4159cf20003, negotiated timeout = 10000
[junit4:junit4]   2> 14290 T1084 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@355f70 name:ZooKeeperConnection Watcher:127.0.0.1:58248/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14290 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14291 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 14294 T1076 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159cf20003 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 14294 T982 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 14296 T982 oasc.Overseer.start Overseer (id=88563555421519875-127.0.0.1:58248_solr-n_0000000000) starting
[junit4:junit4]   2> 14296 T1076 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159cf20003 type:create cxid:0xe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14297 T1076 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159cf20003 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14298 T1076 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159cf20003 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14298 T1076 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159cf20003 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14299 T1086 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 14299 T1076 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159cf20000 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14300 T1076 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159cf20000 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 14303 T1085 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14303 T1085 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4:junit4]   2> 14304 T1085 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard":"s1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "roles":""}
[junit4:junit4]   2> 14305 T1078 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 14306 T1085 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core2",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard":"s1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "roles":""}
[junit4:junit4]   2> 14307 T1078 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 14308 T1085 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 14311 T1085 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14311 T1085 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core3",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard":"s1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1/solr",
[junit4:junit4]   2> 	  "roles":""}
[junit4:junit4]   2> 14314 T1078 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (0)
[junit4:junit4]   2> 14354 T1055 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 14403 T1076 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159cf20003
[junit4:junit4]   2> 14404 T1084 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14404 T1073 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59365 which had sessionid 0x13aa4159cf20003
[junit4:junit4]   2> 14404 T982 oaz.ZooKeeper.close Session: 0x13aa4159cf20003 closed
[junit4:junit4]   2> 14405 T1076 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159cf20000
[junit4:junit4]   2> 14406 T1073 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:59362 which had sessionid 0x13aa4159cf20000
[junit4:junit4]   2> 14406 T1078 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14406 T982 oaz.ZooKeeper.close Session: 0x13aa4159cf20000 closed
[junit4:junit4]   2> 14407 T1075 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 14407 T1076 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 14407 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 14409 T1073 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 14410 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 14410 T982 oas.SolrTestCaseJ4.tearDown ###Ending testReplay
[junit4:junit4]   2> 14416 T982 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
[junit4:junit4]   2> 14417 T982 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 14418 T1087 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 14418 T1087 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 14427 T1087 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351372596646/zookeeper/server1/data/version-2/snapshot.196
[junit4:junit4]   2> 14431 T1087 oazsp.FileTxnSnapLog.save Snapshotting: 1cd
[junit4:junit4]   2> 14518 T982 oasc.ZkTestServer.run start zk server on port:56430
[junit4:junit4]   2> 14518 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56430/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@124283c
[junit4:junit4]   2> 14519 T1092 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56430
[junit4:junit4]   2> 14519 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14520 T1092 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56430, initiating session
[junit4:junit4]   2> 14520 T1088 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40015
[junit4:junit4]   2> 14520 T1088 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40015
[junit4:junit4]   2> 14520 T1090 oazsp.FileTxnLog.append Creating new log file: log.1ce
[junit4:junit4]   2> 14522 T1090 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159e060000 with negotiated timeout 10000 for client /127.0.0.1:40015
[junit4:junit4]   2> 14522 T1092 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56430, sessionid = 0x13aa4159e060000, negotiated timeout = 10000
[junit4:junit4]   2> 14522 T1093 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@124283c name:ZooKeeperConnection Watcher:127.0.0.1:56430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14523 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14523 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56430 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1fbd1ea
[junit4:junit4]   2> 14524 T1094 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56430
[junit4:junit4]   2> 14524 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14525 T1088 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40016
[junit4:junit4]   2> 14525 T1094 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56430, initiating session
[junit4:junit4]   2> 14525 T1088 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40016
[junit4:junit4]   2> 14527 T1090 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159e060001 with negotiated timeout 10000 for client /127.0.0.1:40016
[junit4:junit4]   2> 14527 T1094 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56430, sessionid = 0x13aa4159e060001, negotiated timeout = 10000
[junit4:junit4]   2> 14527 T1095 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fbd1ea name:ZooKeeperConnection Watcher:127.0.0.1:56430 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14528 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14534 T1091 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159e060001
[junit4:junit4]   2> 14535 T1088 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40016 which had sessionid 0x13aa4159e060001
[junit4:junit4]   2> 14535 T982 oaz.ZooKeeper.close Session: 0x13aa4159e060001 closed
[junit4:junit4]   2> 14535 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56430 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1ae06f0
[junit4:junit4]   2> 14536 T1095 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14536 T1096 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56430
[junit4:junit4]   2> 14536 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14537 T1088 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40017
[junit4:junit4]   2> 14537 T1096 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56430, initiating session
[junit4:junit4]   2> 14537 T1088 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40017
[junit4:junit4]   2> 14538 T1090 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159e060002 with negotiated timeout 10000 for client /127.0.0.1:40017
[junit4:junit4]   2> 14538 T1096 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56430, sessionid = 0x13aa4159e060002, negotiated timeout = 10000
[junit4:junit4]   2> 14538 T1097 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ae06f0 name:ZooKeeperConnection Watcher:127.0.0.1:56430 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14539 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14539 T982 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 14540 T1091 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159e060002
[junit4:junit4]   2> 14541 T1088 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40017 which had sessionid 0x13aa4159e060002
[junit4:junit4]   2> 14541 T1097 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14541 T982 oaz.ZooKeeper.close Session: 0x13aa4159e060002 closed
[junit4:junit4]   2> 14541 T982 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 14543 T1098 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56430/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@125c1d
[junit4:junit4]   2> 14543 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56430/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@15d897f
[junit4:junit4]   2> 14544 T1099 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56430
[junit4:junit4]   2> 14544 T1098 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14544 T1101 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56430
[junit4:junit4]   2> 14545 T1088 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40018
[junit4:junit4]   2> 14545 T1088 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40019
[junit4:junit4]   2> 14547 T1101 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56430, initiating session
[junit4:junit4]   2> 14547 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14547 T1099 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56430, initiating session
[junit4:junit4]   2> 14547 T1088 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40019
[junit4:junit4]   2> 14547 T1088 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40018
[junit4:junit4]   2> 14548 T1090 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159e060003 with negotiated timeout 10000 for client /127.0.0.1:40019
[junit4:junit4]   2> 14548 T1101 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56430, sessionid = 0x13aa4159e060003, negotiated timeout = 10000
[junit4:junit4]   2> 14549 T1102 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15d897f name:ZooKeeperConnection Watcher:127.0.0.1:56430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14549 T1090 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159e060004 with negotiated timeout 10000 for client /127.0.0.1:40018
[junit4:junit4]   2> 14549 T1099 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56430, sessionid = 0x13aa4159e060004, negotiated timeout = 10000
[junit4:junit4]   2> 14549 T1100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@125c1d name:ZooKeeperConnection Watcher:127.0.0.1:56430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14549 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14549 T1098 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14550 T1098 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 14550 T982 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 14551 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 14553 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060004 type:delete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 14553 T1098 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 14554 T982 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4:junit4]   2> 14555 T1098 oasc.Overseer.start Overseer (id=88563555439607812-127.0.0.1:56430_solr-n_0000000000) starting
[junit4:junit4]   2> 14556 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060004 type:create cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14556 T1102 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 14556 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060003 type:create cxid:0xd zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14557 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060004 type:create cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14557 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060003 type:create cxid:0xe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 14557 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060004 type:create cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14558 T1104 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 14558 T1098 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4:junit4]   2> 14558 T1091 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159e060004
[junit4:junit4]   2> 14559 T1088 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40018 which had sessionid 0x13aa4159e060004
[junit4:junit4]   2> 14559 T1100 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 14559 T1098 oaz.ZooKeeper.close Session: 0x13aa4159e060004 closed
[junit4:junit4]   2> 14560 T1098 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56430/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@f3bc82
[junit4:junit4]   2> 14560 T1105 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56430
[junit4:junit4]   2> 14560 T1098 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 14561 T1088 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40020
[junit4:junit4]   2> 14561 T1105 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56430, initiating session
[junit4:junit4]   2> 14561 T1088 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40020
[junit4:junit4]   2> 14562 T1090 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159e060005 with negotiated timeout 10000 for client /127.0.0.1:40020
[junit4:junit4]   2> 14562 T1105 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56430, sessionid = 0x13aa4159e060005, negotiated timeout = 10000
[junit4:junit4]   2> 14563 T1106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f3bc82 name:ZooKeeperConnection Watcher:127.0.0.1:56430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 14563 T1098 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 14564 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 14565 T1098 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 14566 T1098 oasc.Overseer.start Overseer (id=88563555439607813-127.0.0.1:56430_solr-n_0000000002) starting
[junit4:junit4]   2> 14566 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060005 type:create cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14567 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060005 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14567 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060005 type:create cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14568 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060005 type:create cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 14568 T1108 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 14570 T1107 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 14572 T1107 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14573 T1107 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"state1",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 14573 T1107 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 14573 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060005 type:create cxid:0x18 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 14577 T1102 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 14816 T1085 oasc.Overseer$ClusterStateUpdater.amILeader WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:192)
[junit4:junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:116)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 14816 T1085 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=88563555421519875-127.0.0.1:58248_solr-n_0000000000) am no longer a leader.
[junit4:junit4]   2> 15059 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 15065 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060003 type:delete cxid:0x1f zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 15067 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 15070 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060003 type:create cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15071 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060003 type:create cxid:0x29 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15072 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 15073 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060003 type:create cxid:0x30 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 15080 T982 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56430/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@833e50
[junit4:junit4]   2> 15081 T1107 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 15082 T1109 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:56430
[junit4:junit4]   2> 15083 T1088 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /127.0.0.1:40021
[junit4:junit4]   2> 15084 T982 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 15084 T1109 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:56430, initiating session
[junit4:junit4]   2> 15084 T1107 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"state2",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 15085 T1088 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /127.0.0.1:40021
[junit4:junit4]   2> 15086 T1109 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost.localdomain/127.0.0.1:56430, sessionid = 0x13aa4159e060006, negotiated timeout = 10000
[junit4:junit4]   2> 15086 T1090 oazs.NIOServerCnxn.finishSessionInit Established session 0x13aa4159e060006 with negotiated timeout 10000 for client /127.0.0.1:40021
[junit4:junit4]   2> 15086 T1110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@833e50 name:ZooKeeperConnection Watcher:127.0.0.1:56430/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 15086 T982 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 15087 T982 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 15088 T1102 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 15088 T982 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4:junit4]   2> 15088 T1110 oaz.ClientCnxn$EventThread.processEvent SEVERE Error while calling watcher  java.lang.NullPointerException
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader$2.process(ZkStateReader.java:189)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 15090 T1110 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 15090 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060003 type:create cxid:0x36 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15090 T1102 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 15091 T982 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 15092 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060003 type:create cxid:0x3d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperErrorCode = NodeExists for /solr/collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 15094 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 15094 T1091 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13aa4159e060006 type:create cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15140 T1068 oasc.Overseer$ClusterStateUpdater.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 15354 T1074 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 15559 T1103 oasc.Overseer$ClusterStateUpdater.run WARNING Solr cannot talk to ZK
[junit4:junit4]   2> 15559 T1104 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 15591 T1107 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 15592 T1107 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"state1",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node1",
[junit4:junit4]   2> 	  "base_url":"http://node1/solr/"}
[junit4:junit4]   2> 15594 T1107 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "state":"state2",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core":"core4",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"node2",
[junit4:junit4]   2> 	  "base_url":"http://node2/solr/"}
[junit4:junit4]   2> 15596 T1102 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 75115 T1091 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159e060003
[junit4:junit4]   2> 75115 T1102 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 75115 T1110 oascc.ZkStateReader$3.process Updating live nodes... (1)
[junit4:junit4]   2> 75116 T1088 oazs.NIOServerCnxn.doIO WARNING EndOfStreamException: Unable to read additional data from client sessionid 0x13aa4159e060003, likely client has closed socket
[junit4:junit4]   2> 75116 T982 oaz.ZooKeeper.close Session: 0x13aa4159e060003 closed
[junit4:junit4]   2> 75116 T1088 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40019 which had sessionid 0x13aa4159e060003
[junit4:junit4]   2> 75117 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75218 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75319 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75420 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75522 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75623 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75724 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75825 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75927 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76028 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76116 T1102 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 76116 T1102 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 76117 T1102 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 76130 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76231 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76332 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76433 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76535 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76636 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76738 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76839 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76940 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77042 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77143 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77244 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77345 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77447 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77548 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77650 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77751 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77853 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 77955 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78056 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78157 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78258 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78359 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78461 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78562 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78664 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78765 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78866 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78967 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79068 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79169 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79270 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79372 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79473 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79574 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79676 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79777 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79878 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79979 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80081 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80182 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80284 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80385 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80487 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80588 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80689 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80790 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80892 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 80993 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81094 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81195 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81296 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81397 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81499 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81600 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81701 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81802 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81903 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82005 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82106 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82207 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82309 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82410 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82511 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82613 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82714 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82815 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82917 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83018 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83119 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83220 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83321 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83423 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83524 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83625 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83727 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83828 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 83930 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84031 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84133 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84234 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84335 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84436 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84538 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84639 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84740 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84842 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84943 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85044 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85145 T982 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85256 T1091 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159e060005
[junit4:junit4]   2> 85256 T1098 oaz.ZooKeeper.close Session: 0x13aa4159e060005 closed
[junit4:junit4]   2> 85256 T1106 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 85257 T1088 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40020 which had sessionid 0x13aa4159e060005
[junit4:junit4]   2> 85257 T1091 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159e060006
[junit4:junit4]   2> 85258 T1110 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 85258 T1088 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40021 which had sessionid 0x13aa4159e060006
[junit4:junit4]   2> 85258 T1110 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 85258 T982 oaz.ZooKeeper.close Session: 0x13aa4159e060006 closed
[junit4:junit4]   2> 85259 T1091 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa4159e060000
[junit4:junit4]   2> 85259 T1093 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 85259 T982 oaz.ZooKeeper.close Session: 0x13aa4159e060000 closed
[junit4:junit4]   2> 85259 T1088 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40015 which had sessionid 0x13aa4159e060000
[junit4:junit4]   2> 85260 T1090 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 85260 T1091 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 85260 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 85262 T1088 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 85262 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 85262 T982 oas.SolrTestCaseJ4.tearDown ###Ending testShardLeaderChange
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTest -Dtests.method=testShardLeaderChange -Dtests.seed=72F707B83BC34278 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hu_HU -Dtests.timezone=America/Bahia_Banderas -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 70.9s J1 | OverseerTest.testShardLeaderChange <<<
[junit4:junit4]    > Throwable #1: org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard:shard1 expected:<core[4]> but was:<core[1]>
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([72F707B83BC34278:ACA4804F215BB789]:0)
[junit4:junit4]    > 	at org.junit.Assert.assertEquals(Assert.java:125)
[junit4:junit4]    > 	at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.java:493)
[junit4:junit4]    > 	at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest.java:652)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:601)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 85307 T982 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignmentBigger
[junit4:junit4]   2> 85307 T982 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 85308 T1111 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 85308 T1111 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 85309 T1111 oazsp.FileSnap.deserialize Reading snapshot /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1351372596646/zoo

[...truncated too long message...]

junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 88633 T1112 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40855 which had sessionid 0x13aa416b2e70042
[junit4:junit4]   2> 88633 T982 oaz.ZooKeeper.close Session: 0x13aa416b2e70042 closed
[junit4:junit4]   2> 88633 T1251 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 88634 T1115 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13aa416b2e70043
[junit4:junit4]   2> 88634 T1253 oascc.ZkStateReader$3.process Updating live nodes... (0)
[junit4:junit4]   2> 88634 T1253 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard11/election
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1249)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:227)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:224)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:83)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 88634 T982 oaz.ZooKeeper.close Session: 0x13aa416b2e70043 closed
[junit4:junit4]   2> 88634 T1112 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:40856 which had sessionid 0x13aa416b2e70043
[junit4:junit4]   2> 88635 T1253 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 88635 T1115 oazs.PrepRequestProcessor.run PrepRequestProcessor exited loop!
[junit4:junit4]   2> 88635 T1114 oazs.SyncRequestProcessor.run SyncRequestProcessor exited!
[junit4:junit4]   2> 88636 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 88637 T1112 oazs.NIOServerCnxn$Factory.run NIOServerCnxn factory exited run method
[junit4:junit4]   2> 88638 T982 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   2> 88639 T982 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignmentBigger
[junit4:junit4]   2> 88640 T982 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> 88640 T982 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 89290 T1133 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 89290 T1133 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89355 T1113 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 89505 T1185 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election/88563560084865082-node54_core54-n_0000000000
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:927)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:244)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:116)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 89506 T1185 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 89506 T1185 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 89622 T1239 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:176)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:173)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:173)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:384)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:353)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:340)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:95)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:155)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:99)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:128)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4]   2> 
[junit4:junit4]   2> 89623 T1239 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 89623 T1239 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 91640 T982 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> NOTE: test params are: codec=Lucene3x, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=hu_HU, timezone=America/Bahia_Banderas
[junit4:junit4]   2> NOTE: Linux 3.2.0-32-generic i386/Oracle Corporation 1.7.0_07 (32-bit)/cpus=8,threads=10,free=108197392,total=300941312
[junit4:junit4]   2> NOTE: All tests run in this JVM: [QueryEqualityTest, TestRealTimeGet, TestUpdate, TestArbitraryIndexDir, RecoveryZkTest, TimeZoneUtilsTest, SoftAutoCommitTest, SolrCoreTest, SpellingQueryConverterTest, QueryParsingTest, ConvertedLegacyTest, StandardRequestHandlerTest, TestPhraseSuggestions, PeerSyncTest, FieldMutatingUpdateProcessorTest, TestFunctionQuery, ZkNodePropsTest, TermVectorComponentDistributedTest, TestCollationKeyRangeQueries, HighlighterTest, TestPluginEnable, TestPropInjectDefaults, TestElisionMultitermQuery, TestDFRSimilarityFactory, TestTrie, IndexSchemaRuntimeFieldTest, PrimitiveFieldTypeTest, FileUtilsTest, TestDocumentBuilder, TestAnalyzedSuggestions, TestStressRecovery, ZkSolrClientTest, MultiTermTest, TestStressVersions, SolrIndexConfigTest, PingRequestHandlerTest, SuggesterTSTTest, TestUtils, UUIDFieldTest, TestRemoteStreaming, TestFastOutputStream, TestSolrDeletionPolicy2, TestQueryUtils, TestQuerySenderListener, PolyFieldTest, BadIndexSchemaTest, TestValueSourceCache, DebugComponentTest, IndexReaderFactoryTest, SyncSliceTest, LukeRequestHandlerTest, TestBM25SimilarityFactory, TestBinaryField, ShowFileRequestHandlerTest, OverseerTest]
[junit4:junit4] Completed on J1 in 91.66s, 8 tests, 1 failure <<< FAILURES!

[...truncated 375 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:335: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:178: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/module-build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1159: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 1 failure, 7 ignored (2 assumptions)

Total time: 25 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/jdk1.7.0_07 -server -XX:+UseG1GC
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message