Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 77BEED542 for ; Thu, 8 Nov 2012 18:15:00 +0000 (UTC) Received: (qmail 81557 invoked by uid 500); 8 Nov 2012 18:14:59 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 81337 invoked by uid 500); 8 Nov 2012 18:14:58 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 81330 invoked by uid 99); 8 Nov 2012 18:14:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Nov 2012 18:14:58 +0000 X-ASF-Spam-Status: No, hits=0.0 required=5.0 tests=NORMAL_HTTP_TO_IP,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [188.138.97.18] (HELO mail.sd-datasolutions.de) (188.138.97.18) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Nov 2012 18:14:48 +0000 Received: from serv1 (localhost.localdomain [127.0.0.1]) by mail.sd-datasolutions.de (Postfix) with ESMTP id 4105C14AA06A for ; Thu, 8 Nov 2012 18:14:26 +0000 (UTC) Date: Thu, 8 Nov 2012 18:14:12 +0000 (UTC) From: Policeman Jenkins Server To: dev@lucene.apache.org Message-ID: <183709054.104.1352398466119.JavaMail.jenkins@serv1.sd-datasolutions.de> Subject: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.6.0_37) - Build # 2282 - Failure! MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_103_103521297.1352398452134" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_103_103521297.1352398452134 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Build: http://jenkins.sd-datasolutions.de/job/Lucene-Solr-trunk-Linux/2282/ Java: 32bit/jdk1.6.0_37 -server -XX:+UseParallelGC 1 tests failed. REGRESSION: org.apache.solr.cloud.OverseerTest.testShardLeaderChange Error Message: Unexpected shard leader coll:collection1 shard:shard1 expected: bu= t was: Stack Trace: org.junit.ComparisonFailure: Unexpected shard leader coll:collection1 shard= :shard1 expected: but was: =09at __randomizedtesting.SeedInfo.seed([6192F3A8BF7966FC:BFC1745FA5E1930D]= :0) =09at org.junit.Assert.assertEquals(Assert.java:125) =09at org.apache.solr.cloud.OverseerTest.verifyShardLeader(OverseerTest.jav= a:493) =09at org.apache.solr.cloud.OverseerTest.testShardLeaderChange(OverseerTest= .java:652) =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) =09at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:39) =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:25) =09at java.lang.reflect.Method.invoke(Method.java:597) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized= Runner.java:1559) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(Random= izedRunner.java:79) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Random= izedRunner.java:737) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random= izedRunner.java:773) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Random= izedRunner.java:787) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:53) =09at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRu= leSetupTeardownChained.java:50) =09at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFi= eldCacheSanity.java:51) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRul= e$1.evaluate(SystemPropertiesInvariantRule.java:55) =09at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleT= hreadAndTestName.java:48) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:70) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:48) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:358) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTa= sk(ThreadLeakControl.java:782) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(Threa= dLeakControl.java:442) =09at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran= domizedRunner.java:746) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Random= izedRunner.java:648) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Random= izedRunner.java:682) =09at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Random= izedRunner.java:693) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$= 1.evaluate(SystemPropertiesRestoreRule.java:53) =09at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef= oreAfterRule.java:45) =09at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor= eClassName.java:42) =09at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRul= e$1.evaluate(SystemPropertiesInvariantRule.java:55) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) =09at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth= odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule= AssertionsRequired.java:43) =09at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai= lure.java:48) =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test= RuleIgnoreAfterMaxFailures.java:70) =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg= noreTestSuites.java:55) =09at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St= atementAdapter.java:36) =09at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.= run(ThreadLeakControl.java:358) =09at java.lang.Thread.run(Thread.java:662) Build Log: [...truncated 8113 lines...] [junit4:junit4] Suite: org.apache.solr.cloud.OverseerTest [junit4:junit4] 2> 0 T1143 oas.SolrTestCaseJ4.initCore ####initCore [junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-So= lr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-OverseerTest-1352398= 067346 [junit4:junit4] 2> 0 T1143 oas.SolrTestCaseJ4.initCore ####initCore end [junit4:junit4] 2> 3 T1143 oas.SolrTestCaseJ4.setUp ###Starting testPlace= holders [junit4:junit4] 2> 3 T1143 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4:junit4] 2> 4 T1144 oazs.ZooKeeperServer.setTickTime tickTime set = to 1000 [junit4:junit4] 2> 4 T1144 oazs.NIOServerCnxn$Factory. binding to p= ort 0.0.0.0/0.0.0.0:0 [junit4:junit4] 2> 5 T1144 oazsp.FileTxnSnapLog.save Snapshotting: 0 [junit4:junit4] 2> 103 T1143 oasc.ZkTestServer.run start zk server on por= t:47207 [junit4:junit4] 2> 104 T1143 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:47207/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@8c296d [junit4:junit4] 2> 105 T1149 oaz.ClientCnxn$SendThread.startConnect Openi= ng socket connection to server /127.0.0.1:47207 [junit4:junit4] 2> 105 T1143 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 105 T1149 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost.localdomain/127.0.0.1:47207, initi= ating session [junit4:junit4] 2> 106 T1145 oazs.NIOServerCnxn$Factory.run Accepted sock= et connection from /127.0.0.1:54925 [junit4:junit4] 2> 106 T1145 oazs.NIOServerCnxn.readConnectRequest Client= attempting to establish new session at /127.0.0.1:54925 [junit4:junit4] 2> 106 T1147 oazsp.FileTxnLog.append Creating new log fil= e: log.1 [junit4:junit4] 2> 212 T1147 oazs.NIOServerCnxn.finishSessionInit Establi= shed session 0x13ae134d6970000 with negotiated timeout 10000 for client /12= 7.0.0.1:54925 [junit4:junit4] 2> 212 T1149 oaz.ClientCnxn$SendThread.readConnectResult = Session establishment complete on server localhost.localdomain/127.0.0.1:47= 207, sessionid =3D 0x13ae134d6970000, negotiated timeout =3D 10000 [junit4:junit4] 2> 213 T1150 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@8c296d name:ZooKeeperConnection = Watcher:127.0.0.1:47207/solr got event WatchedEvent state:SyncConnected typ= e:None path:null path:null type:None [junit4:junit4] 2> 213 T1143 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 213 T1143 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:47207 sessionTimeout=3D10000 watcher=3Dorg= .apache.solr.common.cloud.ConnectionManager@1f329ac [junit4:junit4] 2> 214 T1151 oaz.ClientCnxn$SendThread.startConnect Openi= ng socket connection to server /127.0.0.1:47207 [junit4:junit4] 2> 214 T1143 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 215 T1151 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost.localdomain/127.0.0.1:47207, initi= ating session [junit4:junit4] 2> 215 T1145 oazs.NIOServerCnxn$Factory.run Accepted sock= et connection from /127.0.0.1:54926 [junit4:junit4] 2> 215 T1145 oazs.NIOServerCnxn.readConnectRequest Client= attempting to establish new session at /127.0.0.1:54926 [junit4:junit4] 2> 216 T1147 oazs.NIOServerCnxn.finishSessionInit Establi= shed session 0x13ae134d6970001 with negotiated timeout 10000 for client /12= 7.0.0.1:54926 [junit4:junit4] 2> 216 T1151 oaz.ClientCnxn$SendThread.readConnectResult = Session establishment complete on server localhost.localdomain/127.0.0.1:47= 207, sessionid =3D 0x13ae134d6970001, negotiated timeout =3D 10000 [junit4:junit4] 2> 216 T1152 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@1f329ac name:ZooKeeperConnection= Watcher:127.0.0.1:47207 got event WatchedEvent state:SyncConnected type:No= ne path:null path:null type:None [junit4:junit4] 2> 216 T1143 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 217 T1148 oazs.PrepRequestProcessor.pRequest Processed= session termination for sessionid: 0x13ae134d6970001 [junit4:junit4] 2> 217 T1152 oaz.ClientCnxn$EventThread.run EventThread s= hut down [junit4:junit4] 2> 217 T1145 oazs.NIOServerCnxn.closeSock Closed socket c= onnection for client /127.0.0.1:54926 which had sessionid 0x13ae134d6970001 [junit4:junit4] 2> 217 T1143 oaz.ZooKeeper.close Session: 0x13ae134d69700= 01 closed [junit4:junit4] 2> 218 T1143 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:47207 sessionTimeout=3D10000 watcher=3Dorg= .apache.solr.common.cloud.ConnectionManager@1558803 [junit4:junit4] 2> 219 T1153 oaz.ClientCnxn$SendThread.startConnect Openi= ng socket connection to server /127.0.0.1:47207 [junit4:junit4] 2> 219 T1143 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 219 T1145 oazs.NIOServerCnxn$Factory.run Accepted sock= et connection from /127.0.0.1:54927 [junit4:junit4] 2> 219 T1153 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost.localdomain/127.0.0.1:47207, initi= ating session [junit4:junit4] 2> 219 T1145 oazs.NIOServerCnxn.readConnectRequest Client= attempting to establish new session at /127.0.0.1:54927 [junit4:junit4] 2> 220 T1147 oazs.NIOServerCnxn.finishSessionInit Establi= shed session 0x13ae134d6970002 with negotiated timeout 10000 for client /12= 7.0.0.1:54927 [junit4:junit4] 2> 220 T1153 oaz.ClientCnxn$SendThread.readConnectResult = Session establishment complete on server localhost.localdomain/127.0.0.1:47= 207, sessionid =3D 0x13ae134d6970002, negotiated timeout =3D 10000 [junit4:junit4] 2> 220 T1154 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@1558803 name:ZooKeeperConnection= Watcher:127.0.0.1:47207 got event WatchedEvent state:SyncConnected type:No= ne path:null path:null type:None [junit4:junit4] 2> 220 T1143 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 221 T1143 oascc.SolrZkClient.makePath makePath: /solr [junit4:junit4] 2> 222 T1148 oazs.PrepRequestProcessor.pRequest Processed= session termination for sessionid: 0x13ae134d6970002 [junit4:junit4] 2> 223 T1154 oaz.ClientCnxn$EventThread.run EventThread s= hut down [junit4:junit4] 2> 223 T1145 oazs.NIOServerCnxn.closeSock Closed socket c= onnection for client /127.0.0.1:54927 which had sessionid 0x13ae134d6970002 [junit4:junit4] 2> 223 T1143 oaz.ZooKeeper.close Session: 0x13ae134d69700= 02 closed [junit4:junit4] 2> 223 T1143 oascc.SolrZkClient.makePath makePath: /live_= nodes [junit4:junit4] 2> 225 T1143 oascc.SolrZkClient.makePath makePath: /clust= erstate.json [junit4:junit4] 2> 226 T1143 oascc.ZkStateReader.createClusterStateWatche= rsAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 228 T1143 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:47207/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@3039ce [junit4:junit4] 2> 228 T1155 oaz.ClientCnxn$SendThread.startConnect Openi= ng socket connection to server /127.0.0.1:47207 [junit4:junit4] 2> 228 T1143 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 229 T1155 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost.localdomain/127.0.0.1:47207, initi= ating session [junit4:junit4] 2> 229 T1145 oazs.NIOServerCnxn$Factory.run Accepted sock= et connection from /127.0.0.1:54928 [junit4:junit4] 2> 229 T1145 oazs.NIOServerCnxn.readConnectRequest Client= attempting to establish new session at /127.0.0.1:54928 [junit4:junit4] 2> 230 T1147 oazs.NIOServerCnxn.finishSessionInit Establi= shed session 0x13ae134d6970003 with negotiated timeout 10000 for client /12= 7.0.0.1:54928 [junit4:junit4] 2> 230 T1155 oaz.ClientCnxn$SendThread.readConnectResult = Session establishment complete on server localhost.localdomain/127.0.0.1:47= 207, sessionid =3D 0x13ae134d6970003, negotiated timeout =3D 10000 [junit4:junit4] 2> 230 T1156 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@3039ce name:ZooKeeperConnection = Watcher:127.0.0.1:47207/solr got event WatchedEvent state:SyncConnected typ= e:None path:null path:null type:None [junit4:junit4] 2> 230 T1143 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 231 T1143 oascc.ZkStateReader.createClusterStateWatche= rsAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 233 T1143 oascc.SolrZkClient.makePath makePath: /live_= nodes/node1 [junit4:junit4] 2> 235 T1150 oascc.ZkStateReader$3.process Updating live = nodes... (1) [junit4:junit4] 2> 238 T1156 oascc.ZkStateReader$3.process Updating live = nodes... (1) [junit4:junit4] 2> 238 T1143 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:47207/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@729834 [junit4:junit4] 2> 239 T1157 oaz.ClientCnxn$SendThread.startConnect Openi= ng socket connection to server /127.0.0.1:47207 [junit4:junit4] 2> 239 T1143 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 239 T1157 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost.localdomain/127.0.0.1:47207, initi= ating session [junit4:junit4] 2> 239 T1145 oazs.NIOServerCnxn$Factory.run Accepted sock= et connection from /127.0.0.1:54929 [junit4:junit4] 2> 240 T1145 oazs.NIOServerCnxn.readConnectRequest Client= attempting to establish new session at /127.0.0.1:54929 [junit4:junit4] 2> 240 T1157 oaz.ClientCnxn$SendThread.readConnectResult = Session establishment complete on server localhost.localdomain/127.0.0.1:47= 207, sessionid =3D 0x13ae134d6970004, negotiated timeout =3D 10000 [junit4:junit4] 2> 240 T1147 oazs.NIOServerCnxn.finishSessionInit Establi= shed session 0x13ae134d6970004 with negotiated timeout 10000 for client /12= 7.0.0.1:54929 [junit4:junit4] 2> 240 T1158 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@729834 name:ZooKeeperConnection = Watcher:127.0.0.1:47207/solr got event WatchedEvent state:SyncConnected typ= e:None path:null path:null type:None [junit4:junit4] 2> 241 T1143 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 242 T1143 oascc.SolrZkClient.makePath makePath: /overs= eer_elect/election [junit4:junit4] 2> 245 T1148 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13ae134d6970004 type:dele= te cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/over= seer_elect/leader [junit4:junit4] 2> 245 T1143 oascc.SolrZkClient.makePath makePath: /overs= eer_elect/leader [junit4:junit4] 2> 247 T1143 oasc.Overseer.start Overseer (id=3D886307597= 41915140-127.0.0.1:47207_solr-n_0000000000) starting [junit4:junit4] 2> 248 T1148 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13ae134d6970004 type:crea= te cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 249 T1148 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13ae134d6970004 type:crea= te cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 249 T1148 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13ae134d6970004 type:crea= te cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 250 T1160 oasc.OverseerCollectionProcessor.run Process= current queue of collection messages [junit4:junit4] 2> 250 T1148 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13ae134d6970003 type:crea= te cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 251 T1159 oasc.Overseer$ClusterStateUpdater.run Starti= ng to work on the main queue [junit4:junit4] 2> 251 T1148 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13ae134d6970003 type:crea= te cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:= /solr/overseer/queue Error:KeeperErrorCode =3D NoNode for /solr/overseer/qu= eue [junit4:junit4] 2> 754 T1159 oascc.ZkStateReader.updateClusterState Updat= ing cloud state from ZooKeeper...=20 [junit4:junit4] 2> 754 T1159 oasc.Overseer$ClusterStateUpdater.updateStat= e Update state numShards=3D12 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"12", [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://node1/solr/"} [junit4:junit4] 2> 754 T1159 oasc.Overseer$ClusterStateUpdater.createColl= ection Create collection collection1 with numShards 12 [junit4:junit4] 2> 755 T1148 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13ae134d6970004 type:crea= te cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer/queue-work Error:KeeperErrorCode =3D NoNode for /solr/overs= eer/queue-work [junit4:junit4] 2> 758 T1150 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (1) [junit4:junit4] 2> 758 T1156 oascc.ZkStateReader$2.process A cluster stat= e change has occurred - updating... (1) [junit4:junit4] 2> 1252 T1143 oascc.SolrZkClient.makePath makePath: /coll= ections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 1259 T1148 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134d6970003 type:del= ete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/collections/collection1/leaders Error:KeeperErrorCode =3D NoNode fo= r /solr/collections/collection1/leaders [junit4:junit4] 2> 1259 T1143 oascc.SolrZkClient.makePath makePath: /coll= ections/collection1/leaders/shard1 [junit4:junit4] 2> 1262 T1148 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134d6970003 type:cre= ate cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1263 T1143 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 1264 T1148 oazs.PrepRequestProcessor.pRequest Processe= d session termination for sessionid: 0x13ae134d6970004 [junit4:junit4] 2> 1264 T1158 oaz.ClientCnxn$EventThread.run EventThread = shut down [junit4:junit4] 2> 1264 T1145 oazs.NIOServerCnxn.closeSock Closed socket = connection for client /127.0.0.1:54929 which had sessionid 0x13ae134d697000= 4 [junit4:junit4] 2> 1264 T1143 oaz.ZooKeeper.close Session: 0x13ae134d6970= 004 closed [junit4:junit4] 2> 1265 T1148 oazs.PrepRequestProcessor.pRequest Processe= d session termination for sessionid: 0x13ae134d6970003 [junit4:junit4] 2> 1265 T1150 oascc.ZkStateReader$3.process Updating live= nodes... (0) [junit4:junit4] 2> 1266 T1156 oascc.ZkStateReader$3.process Updating live= nodes... (0) [junit4:junit4] 2> 1266 T1150 oascc.ZkStateReader$2.process A cluster sta= te change has occurred - updating... (0) [junit4:junit4] 2> 1266 T1156 oascc.ZkStateReader$2.process A cluster sta= te change has occurred - updating... (0) [junit4:junit4] 2> 1266 T1145 oazs.NIOServerCnxn.closeSock Closed socket = connection for client /127.0.0.1:54928 which had sessionid 0x13ae134d697000= 3 [junit4:junit4] 2> 1266 T1143 oaz.ZooKeeper.close Session: 0x13ae134d6970= 003 closed [junit4:junit4] 2> 1267 T1148 oazs.PrepRequestProcessor.pRequest Processe= d session termination for sessionid: 0x13ae134d6970000 [junit4:junit4] 2> 1267 T1150 oaz.ClientCnxn$EventThread.run EventThread = shut down [junit4:junit4] 2> 1267 T1145 oazs.NIOServerCnxn.closeSock Closed socket = connection for client /127.0.0.1:54925 which had sessionid 0x13ae134d697000= 0 [junit4:junit4] 2> 1267 T1143 oaz.ZooKeeper.close Session: 0x13ae134d6970= 000 closed [junit4:junit4] 2> 1267 T1148 oazs.PrepRequestProcessor.run PrepRequestPr= ocessor exited loop! [junit4:junit4] 2> 1267 T1147 oazs.SyncRequestProcessor.run SyncRequestPr= ocessor exited! [junit4:junit4] 2> 1268 T1143 oazs.FinalRequestProcessor.shutdown shutdow= n of request processor complete [junit4:junit4] 2> 1269 T1145 oazs.NIOServerCnxn$Factory.run NIOServerCnx= n factory exited run method [junit4:junit4] 2> 1269 T1143 oazs.FinalRequestProcessor.shutdown shutdow= n of request processor complete [junit4:junit4] 2> 1269 T1143 oas.SolrTestCaseJ4.tearDown ###Ending testP= laceholders [junit4:junit4] 2> 1273 T1143 oas.SolrTestCaseJ4.setUp ###Starting testSh= ardLeaderChange [junit4:junit4] 2> 1274 T1143 oasc.ZkTestServer.run STARTING ZK TEST SERV= ER [junit4:junit4] 2> 1274 T1161 oazs.ZooKeeperServer.setTickTime tickTime s= et to 1000 [junit4:junit4] 2> 1274 T1161 oazs.NIOServerCnxn$Factory. binding t= o port 0.0.0.0/0.0.0.0:0 [junit4:junit4] 2> 1275 T1161 oazsp.FileSnap.deserialize Reading snapshot= /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/te= st/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version-= 2/snapshot.0 [junit4:junit4] 2> 1276 T1161 oazsp.FileTxnSnapLog.save Snapshotting: 2e [junit4:junit4] 2> 1374 T1143 oasc.ZkTestServer.run start zk server on po= rt:50631 [junit4:junit4] 2> 1375 T1143 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:50631/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@cf0b27 [junit4:junit4] 2> 1375 T1166 oaz.ClientCnxn$SendThread.startConnect Open= ing socket connection to server /127.0.0.1:50631 [junit4:junit4] 2> 1375 T1143 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 1376 T1166 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:50631, init= iating session [junit4:junit4] 2> 1376 T1162 oazs.NIOServerCnxn$Factory.run Accepted soc= ket connection from /127.0.0.1:45364 [junit4:junit4] 2> 1376 T1162 oazs.NIOServerCnxn.readConnectRequest Clien= t attempting to establish new session at /127.0.0.1:45364 [junit4:junit4] 2> 1376 T1164 oazsp.FileTxnLog.append Creating new log fi= le: log.2f [junit4:junit4] 2> 1378 T1164 oazs.NIOServerCnxn.finishSessionInit Establ= ished session 0x13ae134db8f0000 with negotiated timeout 10000 for client /1= 27.0.0.1:45364 [junit4:junit4] 2> 1378 T1166 oaz.ClientCnxn$SendThread.readConnectResult= Session establishment complete on server localhost.localdomain/127.0.0.1:5= 0631, sessionid =3D 0x13ae134db8f0000, negotiated timeout =3D 10000 [junit4:junit4] 2> 1378 T1167 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@cf0b27 name:ZooKeeperConnection= Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected ty= pe:None path:null path:null type:None [junit4:junit4] 2> 1378 T1143 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 1378 T1143 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:50631 sessionTimeout=3D10000 watcher=3Dor= g.apache.solr.common.cloud.ConnectionManager@11b3588 [junit4:junit4] 2> 1379 T1168 oaz.ClientCnxn$SendThread.startConnect Open= ing socket connection to server /127.0.0.1:50631 [junit4:junit4] 2> 1379 T1143 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 1379 T1168 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:50631, init= iating session [junit4:junit4] 2> 1379 T1162 oazs.NIOServerCnxn$Factory.run Accepted soc= ket connection from /127.0.0.1:45365 [junit4:junit4] 2> 1380 T1162 oazs.NIOServerCnxn.readConnectRequest Clien= t attempting to establish new session at /127.0.0.1:45365 [junit4:junit4] 2> 1380 T1164 oazs.NIOServerCnxn.finishSessionInit Establ= ished session 0x13ae134db8f0001 with negotiated timeout 10000 for client /1= 27.0.0.1:45365 [junit4:junit4] 2> 1380 T1168 oaz.ClientCnxn$SendThread.readConnectResult= Session establishment complete on server localhost.localdomain/127.0.0.1:5= 0631, sessionid =3D 0x13ae134db8f0001, negotiated timeout =3D 10000 [junit4:junit4] 2> 1380 T1169 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@11b3588 name:ZooKeeperConnectio= n Watcher:127.0.0.1:50631 got event WatchedEvent state:SyncConnected type:N= one path:null path:null type:None [junit4:junit4] 2> 1381 T1143 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 1389 T1165 oazs.PrepRequestProcessor.pRequest Processe= d session termination for sessionid: 0x13ae134db8f0001 [junit4:junit4] 2> 1390 T1169 oaz.ClientCnxn$EventThread.run EventThread = shut down [junit4:junit4] 2> 1390 T1162 oazs.NIOServerCnxn.closeSock Closed socket = connection for client /127.0.0.1:45365 which had sessionid 0x13ae134db8f000= 1 [junit4:junit4] 2> 1390 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f0= 001 closed [junit4:junit4] 2> 1391 T1143 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:50631 sessionTimeout=3D10000 watcher=3Dor= g.apache.solr.common.cloud.ConnectionManager@34c129 [junit4:junit4] 2> 1391 T1170 oaz.ClientCnxn$SendThread.startConnect Open= ing socket connection to server /127.0.0.1:50631 [junit4:junit4] 2> 1391 T1143 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 1391 T1170 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:50631, init= iating session [junit4:junit4] 2> 1392 T1162 oazs.NIOServerCnxn$Factory.run Accepted soc= ket connection from /127.0.0.1:45366 [junit4:junit4] 2> 1392 T1162 oazs.NIOServerCnxn.readConnectRequest Clien= t attempting to establish new session at /127.0.0.1:45366 [junit4:junit4] 2> 1392 T1164 oazs.NIOServerCnxn.finishSessionInit Establ= ished session 0x13ae134db8f0002 with negotiated timeout 10000 for client /1= 27.0.0.1:45366 [junit4:junit4] 2> 1392 T1170 oaz.ClientCnxn$SendThread.readConnectResult= Session establishment complete on server localhost.localdomain/127.0.0.1:5= 0631, sessionid =3D 0x13ae134db8f0002, negotiated timeout =3D 10000 [junit4:junit4] 2> 1393 T1171 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@34c129 name:ZooKeeperConnection= Watcher:127.0.0.1:50631 got event WatchedEvent state:SyncConnected type:No= ne path:null path:null type:None [junit4:junit4] 2> 1393 T1143 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 1393 T1143 oascc.SolrZkClient.makePath makePath: /solr [junit4:junit4] 2> 1394 T1165 oazs.PrepRequestProcessor.pRequest Processe= d session termination for sessionid: 0x13ae134db8f0002 [junit4:junit4] 2> 1395 T1171 oaz.ClientCnxn$EventThread.run EventThread = shut down [junit4:junit4] 2> 1395 T1162 oazs.NIOServerCnxn.closeSock Closed socket = connection for client /127.0.0.1:45366 which had sessionid 0x13ae134db8f000= 2 [junit4:junit4] 2> 1395 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f0= 002 closed [junit4:junit4] 2> 1395 T1143 oascc.SolrZkClient.makePath makePath: /live= _nodes [junit4:junit4] 2> 1397 T1172 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:50631/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@1333f77 [junit4:junit4] 2> 1397 T1143 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:50631/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@18bd49c [junit4:junit4] 2> 1397 T1173 oaz.ClientCnxn$SendThread.startConnect Open= ing socket connection to server /127.0.0.1:50631 [junit4:junit4] 2> 1397 T1172 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 1404 T1175 oaz.ClientCnxn$SendThread.startConnect Open= ing socket connection to server /127.0.0.1:50631 [junit4:junit4] 2> 1404 T1143 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 1405 T1162 oazs.NIOServerCnxn$Factory.run Accepted soc= ket connection from /127.0.0.1:45367 [junit4:junit4] 2> 1405 T1173 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:50631, init= iating session [junit4:junit4] 2> 1405 T1175 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:50631, init= iating session [junit4:junit4] 2> 1405 T1162 oazs.NIOServerCnxn$Factory.run Accepted soc= ket connection from /127.0.0.1:45368 [junit4:junit4] 2> 1406 T1162 oazs.NIOServerCnxn.readConnectRequest Clien= t attempting to establish new session at /127.0.0.1:45367 [junit4:junit4] 2> 1406 T1162 oazs.NIOServerCnxn.readConnectRequest Clien= t attempting to establish new session at /127.0.0.1:45368 [junit4:junit4] 2> 1406 T1164 oazs.NIOServerCnxn.finishSessionInit Establ= ished session 0x13ae134db8f0003 with negotiated timeout 10000 for client /1= 27.0.0.1:45367 [junit4:junit4] 2> 1407 T1173 oaz.ClientCnxn$SendThread.readConnectResult= Session establishment complete on server localhost.localdomain/127.0.0.1:5= 0631, sessionid =3D 0x13ae134db8f0003, negotiated timeout =3D 10000 [junit4:junit4] 2> 1407 T1174 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@1333f77 name:ZooKeeperConnectio= n Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 1407 T1172 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 1407 T1175 oaz.ClientCnxn$SendThread.readConnectResult= Session establishment complete on server localhost.localdomain/127.0.0.1:5= 0631, sessionid =3D 0x13ae134db8f0004, negotiated timeout =3D 10000 [junit4:junit4] 2> 1407 T1164 oazs.NIOServerCnxn.finishSessionInit Establ= ished session 0x13ae134db8f0004 with negotiated timeout 10000 for client /1= 27.0.0.1:45368 [junit4:junit4] 2> 1407 T1176 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@18bd49c name:ZooKeeperConnectio= n Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 1408 T1172 oascc.SolrZkClient.makePath makePath: /over= seer_elect/election [junit4:junit4] 2> 1408 T1143 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 1408 T1143 oascc.SolrZkClient.makePath makePath: /clus= terstate.json [junit4:junit4] 2> 1410 T1143 oascc.ZkStateReader.createClusterStateWatch= ersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 1410 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0003 type:del= ete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/ove= rseer_elect/leader [junit4:junit4] 2> 1410 T1172 oascc.SolrZkClient.makePath makePath: /over= seer_elect/leader [junit4:junit4] 2> 1411 T1143 oascc.SolrZkClient.makePath makePath: /live= _nodes/node1 [junit4:junit4] 2> 1412 T1172 oasc.Overseer.start Overseer (id=3D88630759= 825276931-127.0.0.1:50631_solr-n_0000000000) starting [junit4:junit4] 2> 1414 T1176 oascc.ZkStateReader$3.process Updating live= nodes... (1) [junit4:junit4] 2> 1414 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0003 type:cre= ate cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1414 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0004 type:cre= ate cxid:0xd zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1415 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0003 type:cre= ate cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1415 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0004 type:cre= ate cxid:0xe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer/queue Error:KeeperErrorCode =3D NoNode for /solr/overseer/q= ueue [junit4:junit4] 2> 1415 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0003 type:cre= ate cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1416 T1178 oasc.OverseerCollectionProcessor.run Proces= s current queue of collection messages [junit4:junit4] 2> 1416 T1172 oasc.OverseerTest$OverseerRestarter.run Kil= ling overseer. [junit4:junit4] 2> 1416 T1165 oazs.PrepRequestProcessor.pRequest Processe= d session termination for sessionid: 0x13ae134db8f0003 [junit4:junit4] 2> 1416 T1174 oaz.ClientCnxn$EventThread.run EventThread = shut down [junit4:junit4] 2> 1417 T1162 oazs.NIOServerCnxn.closeSock Closed socket = connection for client /127.0.0.1:45367 which had sessionid 0x13ae134db8f000= 3 [junit4:junit4] 2> 1416 T1172 oaz.ZooKeeper.close Session: 0x13ae134db8f0= 003 closed [junit4:junit4] 2> 1417 T1172 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:50631/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@1ee1fda [junit4:junit4] 2> 1417 T1179 oaz.ClientCnxn$SendThread.startConnect Open= ing socket connection to server /127.0.0.1:50631 [junit4:junit4] 2> 1417 T1172 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 1418 T1179 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:50631, init= iating session [junit4:junit4] 2> 1418 T1162 oazs.NIOServerCnxn$Factory.run Accepted soc= ket connection from /127.0.0.1:45369 [junit4:junit4] 2> 1418 T1162 oazs.NIOServerCnxn.readConnectRequest Clien= t attempting to establish new session at /127.0.0.1:45369 [junit4:junit4] 2> 1419 T1164 oazs.NIOServerCnxn.finishSessionInit Establ= ished session 0x13ae134db8f0005 with negotiated timeout 10000 for client /1= 27.0.0.1:45369 [junit4:junit4] 2> 1419 T1179 oaz.ClientCnxn$SendThread.readConnectResult= Session establishment complete on server localhost.localdomain/127.0.0.1:5= 0631, sessionid =3D 0x13ae134db8f0005, negotiated timeout =3D 10000 [junit4:junit4] 2> 1420 T1180 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@1ee1fda name:ZooKeeperConnectio= n Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 1420 T1172 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 1421 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0005 type:del= ete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/ove= rseer_elect/leader [junit4:junit4] 2> 1421 T1172 oascc.SolrZkClient.makePath makePath: /over= seer_elect/leader [junit4:junit4] 2> 1422 T1172 oasc.Overseer.start Overseer (id=3D88630759= 825276933-127.0.0.1:50631_solr-n_0000000002) starting [junit4:junit4] 2> 1423 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0005 type:cre= ate cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1423 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0005 type:cre= ate cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1424 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0005 type:cre= ate cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1424 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0005 type:cre= ate cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1425 T1182 oasc.OverseerCollectionProcessor.run Proces= s current queue of collection messages [junit4:junit4] 2> 1425 T1181 oasc.Overseer$ClusterStateUpdater.run Start= ing to work on the main queue [junit4:junit4] 2> 1426 T1181 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 1426 T1181 oasc.Overseer$ClusterStateUpdater.updateSta= te Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "state":"state1", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://node1/solr/"} [junit4:junit4] 2> 1427 T1181 oasc.Overseer$ClusterStateUpdater.createCol= lection Create collection collection1 with numShards 1 [junit4:junit4] 2> 1427 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0005 type:cre= ate cxid:0x18 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer/queue-work Error:KeeperErrorCode =3D NoNode for /solr/over= seer/queue-work [junit4:junit4] 2> 1430 T1176 oascc.ZkStateReader$2.process A cluster sta= te change has occurred - updating... (1) [junit4:junit4] 2> 1655 T1146 oazs.SessionTrackerImpl.run SessionTrackerI= mpl exited loop! [junit4:junit4] 2> 1760 T1159 oasc.Overseer$ClusterStateUpdater.amILeader= WARNING org.apache.zookeeper.KeeperException$SessionExpiredException: Kee= perErrorCode =3D Session expired for /overseer_elect/leader [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper= .java:927) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:244) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getDat= a(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.amILeader(Overseer.java:192) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.run(Overseer.java:116) [junit4:junit4] 2> =09at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2>=20 [junit4:junit4] 2> 1762 T1159 oasc.Overseer$ClusterStateUpdater.amILeader= According to ZK I (id=3D88630759741915140-127.0.0.1:47207_solr-n_000000000= 0) am no longer a leader. [junit4:junit4] 2> 1917 T1143 oascc.SolrZkClient.makePath makePath: /coll= ections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 1922 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0004 type:del= ete cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/collections/collection1/leaders Error:KeeperErrorCode =3D NoNode fo= r /solr/collections/collection1/leaders [junit4:junit4] 2> 1923 T1143 oascc.SolrZkClient.makePath makePath: /coll= ections/collection1/leaders/shard1 [junit4:junit4] 2> 1928 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0004 type:cre= ate cxid:0x28 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1929 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0004 type:cre= ate cxid:0x2a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1930 T1143 oascc.SolrZkClient.makePath makePath: /coll= ections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 1932 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0004 type:cre= ate cxid:0x31 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperEr= rorCode =3D NodeExists for /solr/collections/collection1/leader_elect/shard= 1/election [junit4:junit4] 2> 1932 T1181 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 1933 T1143 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:50631/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@8d844b [junit4:junit4] 2> 1934 T1183 oaz.ClientCnxn$SendThread.startConnect Open= ing socket connection to server /127.0.0.1:50631 [junit4:junit4] 2> 1934 T1143 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 1935 T1183 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:50631, init= iating session [junit4:junit4] 2> 1935 T1162 oazs.NIOServerCnxn$Factory.run Accepted soc= ket connection from /127.0.0.1:45370 [junit4:junit4] 2> 1935 T1181 oasc.Overseer$ClusterStateUpdater.updateSta= te Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "state":"state2", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://node1/solr/"} [junit4:junit4] 2> 1935 T1162 oazs.NIOServerCnxn.readConnectRequest Clien= t attempting to establish new session at /127.0.0.1:45370 [junit4:junit4] 2> 1936 T1164 oazs.NIOServerCnxn.finishSessionInit Establ= ished session 0x13ae134db8f0006 with negotiated timeout 10000 for client /1= 27.0.0.1:45370 [junit4:junit4] 2> 1936 T1183 oaz.ClientCnxn$SendThread.readConnectResult= Session establishment complete on server localhost.localdomain/127.0.0.1:5= 0631, sessionid =3D 0x13ae134db8f0006, negotiated timeout =3D 10000 [junit4:junit4] 2> 1936 T1184 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@8d844b name:ZooKeeperConnection= Watcher:127.0.0.1:50631/solr got event WatchedEvent state:SyncConnected ty= pe:None path:null path:null type:None [junit4:junit4] 2> 1936 T1143 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 1937 T1143 oascc.ZkStateReader.createClusterStateWatch= ersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 2218 T1176 oascc.ZkStateReader$2.process A cluster sta= te change has occurred - updating... (1) [junit4:junit4] 2> 2218 T1184 oaz.ClientCnxn$EventThread.processEvent SEV= ERE Error while calling watcher java.lang.NullPointerException [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkStateReader$2.pro= cess(ZkStateReader.java:189) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.proc= essEvent(ClientCnxn.java:526) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(= ClientCnxn.java:502) [junit4:junit4] 2>=20 [junit4:junit4] 2> 2218 T1143 oascc.SolrZkClient.makePath makePath: /live= _nodes/node2 [junit4:junit4] 2> 2221 T1176 oascc.ZkStateReader$2.process A cluster sta= te change has occurred - updating... (1) [junit4:junit4] 2> 2222 T1184 oascc.ZkStateReader$3.process Updating live= nodes... (2) [junit4:junit4] 2> 2222 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0004 type:cre= ate cxid:0x3a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 2222 T1176 oascc.ZkStateReader$3.process Updating live= nodes... (2) [junit4:junit4] 2> 2223 T1143 oascc.SolrZkClient.makePath makePath: /coll= ections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 2224 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0004 type:cre= ate cxid:0x41 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperEr= rorCode =3D NodeExists for /solr/collections/collection1/leader_elect/shard= 1/election [junit4:junit4] 2> 2226 T1143 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 2227 T1165 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13ae134db8f0006 type:cre= ate cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path= :/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 2266 T1156 oascc.ZkStateReader$2.process WARNING ZooKe= eper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 2266 T1156 oaz.ClientCnxn$EventThread.run EventThread = shut down [junit4:junit4] 2> 2416 T1178 oasc.OverseerCollectionProcessor.amILeader = WARNING org.apache.zookeeper.KeeperException$SessionExpiredException: Keep= erErrorCode =3D Session expired for /overseer_elect/leader [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper= .java:927) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:244) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getDat= a(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.cloud.OverseerCollectionProcesso= r.amILeader(OverseerCollectionProcessor.java:129) [junit4:junit4] 2> =09at org.apache.solr.cloud.OverseerCollectionProcesso= r.run(OverseerCollectionProcessor.java:84) [junit4:junit4] 2> =09at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2>=20 [junit4:junit4] 2> 2416 T1177 oasc.Overseer$ClusterStateUpdater.run WARNI= NG Solr cannot talk to ZK [junit4:junit4] 2> 2417 T1178 oasc.OverseerCollectionProcessor.amILeader = According to ZK I (id=3D88630759825276931-127.0.0.1:50631_solr-n_0000000000= ) am no longer a leader. [junit4:junit4] 2> 2723 T1181 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 2723 T1181 oasc.Overseer$ClusterStateUpdater.updateSta= te Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "state":"state1", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://node1/solr/"} [junit4:junit4] 2> 2725 T1181 oasc.Overseer$ClusterStateUpdater.updateSta= te Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "state":"state2", [junit4:junit4] 2> =09 "core":"core4", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node2", [junit4:junit4] 2> =09 "base_url":"http://node2/solr/"} [junit4:junit4] 2> 2727 T1176 oascc.ZkStateReader$2.process A cluster sta= te change has occurred - updating... (2) [junit4:junit4] 2> 62244 T1176 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (2) [junit4:junit4] 2> 62244 T1165 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae134db8f0004 [junit4:junit4] 2> 62245 T1184 oascc.ZkStateReader$3.process Updating liv= e nodes... (1) [junit4:junit4] 2> 62245 T1162 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:45368 which had sessionid 0x13ae134db8f00= 04 [junit4:junit4] 2> 62245 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f= 0004 closed [junit4:junit4] 2> 62246 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 62347 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 62448 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 62550 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 62651 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 62753 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 62855 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 62956 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63057 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63159 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63245 T1176 oascc.ZkStateReader$2.process WARNING ZooK= eeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 63245 T1176 oascc.ZkStateReader$3.process WARNING ZooK= eeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 63245 T1176 oasc.LeaderElector$1.process WARNING org.= apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = =3D Session expired for /collections/collection1/leader_elect/shard1/electi= on [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getChildren(ZooKe= eper.java:1249) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$6.exec= ute(SolrZkClient.java:227) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$6.exec= ute(SolrZkClient.java:224) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getChi= ldren(SolrZkClient.java:224) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.checkIfIamLe= ader(LeaderElector.java:83) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.access$000(L= eaderElector.java:55) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector$1.process(Le= aderElector.java:128) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.proc= essEvent(ClientCnxn.java:526) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(= ClientCnxn.java:502) [junit4:junit4] 2>=20 [junit4:junit4] 2> 63245 T1176 oasc.LeaderElector$1.process WARNING org.= apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = =3D Session expired for /collections/collection1/leader_elect/shard1/electi= on [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getChildren(ZooKe= eper.java:1249) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$6.exec= ute(SolrZkClient.java:227) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$6.exec= ute(SolrZkClient.java:224) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getChi= ldren(SolrZkClient.java:224) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.checkIfIamLe= ader(LeaderElector.java:83) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.access$000(L= eaderElector.java:55) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector$1.process(Le= aderElector.java:128) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.proc= essEvent(ClientCnxn.java:526) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(= ClientCnxn.java:502) [junit4:junit4] 2>=20 [junit4:junit4] 2> 63246 T1176 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 63260 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63361 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63463 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63564 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63665 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63767 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63868 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 63969 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64071 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64172 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64274 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64376 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64477 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64579 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64680 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64782 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64883 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 64984 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65085 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65187 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65288 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65390 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65491 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65592 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65694 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65795 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65897 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 65998 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 66099 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 66201 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 66302 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 66403 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 66505 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 66606 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 66707 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 66809 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 66910 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67011 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67113 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67215 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67316 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67418 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67519 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67621 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67722 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67824 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 67925 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68026 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68128 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68229 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68331 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68432 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68533 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68635 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68736 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68837 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 68938 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69040 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69141 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69243 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69344 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69445 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69547 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69648 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69749 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69851 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 69952 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70054 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70156 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70257 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70359 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70460 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70562 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70664 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70765 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70867 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 70969 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71070 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71171 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71273 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71374 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71475 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71576 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71678 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71779 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71880 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 71982 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72083 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72185 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72286 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72387 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72489 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72590 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72691 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72793 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72894 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 72996 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 73097 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 73198 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 73300 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 73402 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 73503 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 73605 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 73706 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 73808 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 73909 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74010 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74112 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74214 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74315 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74417 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74519 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74620 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74722 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74823 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 74925 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75026 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75128 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75229 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75330 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75432 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75533 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75634 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75736 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75837 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 75939 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76040 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76142 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76243 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76345 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76446 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76547 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76649 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76750 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76852 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 76953 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77055 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77156 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77258 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77359 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77460 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77562 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77663 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77764 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77866 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 77967 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78068 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78169 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78271 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78372 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78474 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78576 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78678 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78779 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78881 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 78982 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79084 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79186 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79287 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79388 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79490 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79592 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79693 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79795 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79896 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 79997 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 80098 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 80203 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 80304 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 80406 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 80507 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 80609 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 80710 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 80811 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 80912 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81014 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81115 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81216 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81318 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81419 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81520 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81622 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81723 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81824 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 81926 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 82027 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 82129 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 82230 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 82332 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 82433 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 82604 T1165 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae134db8f0005 [junit4:junit4] 2> 82605 T1172 oaz.ZooKeeper.close Session: 0x13ae134db8f= 0005 closed [junit4:junit4] 2> 82605 T1180 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 82605 T1162 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:45369 which had sessionid 0x13ae134db8f00= 05 [junit4:junit4] 2> 82606 T1165 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae134db8f0006 [junit4:junit4] 2> 82606 T1184 oascc.ZkStateReader$3.process Updating liv= e nodes... (0) [junit4:junit4] 2> 82607 T1184 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 82607 T1162 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:45370 which had sessionid 0x13ae134db8f00= 06 [junit4:junit4] 2> 82607 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f= 0006 closed [junit4:junit4] 2> 82607 T1165 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae134db8f0000 [junit4:junit4] 2> 82608 T1167 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 82608 T1162 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:45364 which had sessionid 0x13ae134db8f00= 00 [junit4:junit4] 2> 82608 T1143 oaz.ZooKeeper.close Session: 0x13ae134db8f= 0000 closed [junit4:junit4] 2> 82608 T1165 oazs.PrepRequestProcessor.run PrepRequestP= rocessor exited loop! [junit4:junit4] 2> 82608 T1164 oazs.SyncRequestProcessor.run SyncRequestP= rocessor exited! [junit4:junit4] 2> 82609 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 82611 T1162 oazs.NIOServerCnxn$Factory.run NIOServerCn= xn factory exited run method [junit4:junit4] 2> 82611 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 82612 T1143 oas.SolrTestCaseJ4.tearDown ###Ending test= ShardLeaderChange [junit4:junit4] 2> 82654 T1163 oazs.SessionTrackerImpl.run SessionTracker= Impl exited loop! [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=3DOverseerT= est -Dtests.method=3DtestShardLeaderChange -Dtests.seed=3D6192F3A8BF7966FC = -Dtests.multiplier=3D3 -Dtests.slow=3Dtrue -Dtests.locale=3Dnl_NL -Dtests.t= imezone=3DAmerica/Aruba -Dtests.file.encoding=3DUTF-8 [junit4:junit4] FAILURE 81.4s J1 | OverseerTest.testShardLeaderChange <<< [junit4:junit4] > Throwable #1: org.junit.ComparisonFailure: Unexpected = shard leader coll:collection1 shard:shard1 expected: but was: [junit4:junit4] > =09at __randomizedtesting.SeedInfo.seed([6192F3A8BF796= 6FC:BFC1745FA5E1930D]:0) [junit4:junit4] > =09at org.junit.Assert.assertEquals(Assert.java:125) [junit4:junit4] > =09at org.apache.solr.cloud.OverseerTest.verifyShardLe= ader(OverseerTest.java:493) [junit4:junit4] > =09at org.apache.solr.cloud.OverseerTest.testShardLead= erChange(OverseerTest.java:652) [junit4:junit4] > =09at sun.reflect.NativeMethodAccessorImpl.invoke0(Nat= ive Method) [junit4:junit4] > =09at sun.reflect.NativeMethodAccessorImpl.invoke(Nati= veMethodAccessorImpl.java:39) [junit4:junit4] > =09at sun.reflect.DelegatingMethodAccessorImpl.invoke(= DelegatingMethodAccessorImpl.java:25) [junit4:junit4] > =09at java.lang.reflect.Method.invoke(Method.java:597) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.RandomizedRun= ner.invoke(RandomizedRunner.java:1559) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.RandomizedRun= ner.access$600(RandomizedRunner.java:79) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.RandomizedRun= ner$6.evaluate(RandomizedRunner.java:737) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.RandomizedRun= ner$7.evaluate(RandomizedRunner.java:773) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.RandomizedRun= ner$8.evaluate(RandomizedRunner.java:787) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.SystemP= ropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleSetupTeardownChai= ned$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleFieldCacheSanity$= 1.evaluate(TestRuleFieldCacheSanity.java:51) [junit4:junit4] > =09at org.apache.lucene.util.AbstractBeforeAfterRule$1= .evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.SystemP= ropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleThreadAndTestName= $1.evaluate(TestRuleThreadAndTestName.java:48) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFai= lures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleMarkFailure$1.eva= luate(TestRuleMarkFailure.java:48) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.Stateme= ntAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.ThreadLeakCon= trol$StatementRunner.run(ThreadLeakControl.java:358) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.ThreadLeakCon= trol.forkTimeoutingTask(ThreadLeakControl.java:782) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.ThreadLeakCon= trol$3.evaluate(ThreadLeakControl.java:442) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.RandomizedRun= ner.runSingleTest(RandomizedRunner.java:746) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.RandomizedRun= ner$3.evaluate(RandomizedRunner.java:648) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.RandomizedRun= ner$4.evaluate(RandomizedRunner.java:682) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.RandomizedRun= ner$5.evaluate(RandomizedRunner.java:693) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.Stateme= ntAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.SystemP= ropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) [junit4:junit4] > =09at org.apache.lucene.util.AbstractBeforeAfterRule$1= .evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleStoreClassName$1.= evaluate(TestRuleStoreClassName.java:42) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.SystemP= ropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.NoShado= wingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule= .java:39) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.NoShado= wingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule= .java:39) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.Stateme= ntAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleAssertionsRequire= d$1.evaluate(TestRuleAssertionsRequired.java:43) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleMarkFailure$1.eva= luate(TestRuleMarkFailure.java:48) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleIgnoreAfterMaxFai= lures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] > =09at org.apache.lucene.util.TestRuleIgnoreTestSuites$= 1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.rules.Stateme= ntAdapter.evaluate(StatementAdapter.java:36) [junit4:junit4] > =09at com.carrotsearch.randomizedtesting.ThreadLeakCon= trol$StatementRunner.run(ThreadLeakControl.java:358) [junit4:junit4] > =09at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2> 82663 T1143 oas.SolrTestCaseJ4.setUp ###Starting testO= verseerFailure [junit4:junit4] 2> 82663 T1143 oasc.ZkTestServer.run STARTING ZK TEST SER= VER [junit4:junit4] 2> 82664 T1185 oazs.ZooKeeperServer.setTickTime tickTime = set to 1000 [junit4:junit4] 2> 82664 T1185 oazs.NIOServerCnxn$Factory. binding = to port 0.0.0.0/0.0.0.0:0 [junit4:junit4] 2> 82665 T1185 oazsp.FileSnap.deserialize Reading snapsho= t /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/t= est/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version= -2/snapshot.2e [junit4:junit4] 2> 82668 T1185 oazsp.FileTxnSnapLog.save Snapshotting: 91 [junit4:junit4] 2> 82764 T1143 oasc.ZkTestServer.run start zk server on p= ort:59263 [junit4:junit4] 2> 82764 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:59263/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@10d6341 [junit4:junit4] 2> 82765 T1190 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:59263 [junit4:junit4] 2> 82765 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 82765 T1190 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:59263, ini= tiating session [junit4:junit4] 2> 82765 T1186 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:35653 [junit4:junit4] 2> 82766 T1186 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:35653 [junit4:junit4] 2> 82766 T1188 oazsp.FileTxnLog.append Creating new log f= ile: log.92 [junit4:junit4] 2> 82767 T1188 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136197f0000 with negotiated timeout 10000 for client /= 127.0.0.1:35653 [junit4:junit4] 2> 82767 T1190 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 59263, sessionid =3D 0x13ae136197f0000, negotiated timeout =3D 10000 [junit4:junit4] 2> 82768 T1191 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@10d6341 name:ZooKeeperConnecti= on Watcher:127.0.0.1:59263/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 82768 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 82768 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:59263 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@1ac9a93 [junit4:junit4] 2> 82769 T1192 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:59263 [junit4:junit4] 2> 82769 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 82769 T1192 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:59263, ini= tiating session [junit4:junit4] 2> 82769 T1186 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:35654 [junit4:junit4] 2> 82770 T1186 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:35654 [junit4:junit4] 2> 82770 T1188 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136197f0001 with negotiated timeout 10000 for client /= 127.0.0.1:35654 [junit4:junit4] 2> 82770 T1192 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 59263, sessionid =3D 0x13ae136197f0001, negotiated timeout =3D 10000 [junit4:junit4] 2> 82771 T1193 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1ac9a93 name:ZooKeeperConnecti= on Watcher:127.0.0.1:59263 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 82771 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 82779 T1189 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136197f0001 [junit4:junit4] 2> 82780 T1186 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:35654 which had sessionid 0x13ae136197f00= 01 [junit4:junit4] 2> 82780 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f= 0001 closed [junit4:junit4] 2> 82780 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:59263 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@161b30c [junit4:junit4] 2> 82781 T1193 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 82781 T1194 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:59263 [junit4:junit4] 2> 82781 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 82782 T1194 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:59263, ini= tiating session [junit4:junit4] 2> 82782 T1186 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:35655 [junit4:junit4] 2> 82782 T1186 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:35655 [junit4:junit4] 2> 82782 T1188 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136197f0002 with negotiated timeout 10000 for client /= 127.0.0.1:35655 [junit4:junit4] 2> 82782 T1194 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 59263, sessionid =3D 0x13ae136197f0002, negotiated timeout =3D 10000 [junit4:junit4] 2> 82783 T1195 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@161b30c name:ZooKeeperConnecti= on Watcher:127.0.0.1:59263 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 82783 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 82783 T1143 oascc.SolrZkClient.makePath makePath: /sol= r [junit4:junit4] 2> 82785 T1189 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136197f0002 [junit4:junit4] 2> 82785 T1195 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 82785 T1186 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:35655 which had sessionid 0x13ae136197f00= 02 [junit4:junit4] 2> 82785 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f= 0002 closed [junit4:junit4] 2> 82786 T1143 oascc.SolrZkClient.makePath makePath: /liv= e_nodes [junit4:junit4] 2> 82787 T1143 oascc.SolrZkClient.makePath makePath: /clu= sterstate.json [junit4:junit4] 2> 82788 T1143 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 82789 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:59263/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@34054f [junit4:junit4] 2> 82790 T1196 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:59263 [junit4:junit4] 2> 82790 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 82790 T1196 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:59263, ini= tiating session [junit4:junit4] 2> 82790 T1186 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:35656 [junit4:junit4] 2> 82791 T1186 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:35656 [junit4:junit4] 2> 82791 T1188 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136197f0003 with negotiated timeout 10000 for client /= 127.0.0.1:35656 [junit4:junit4] 2> 82791 T1196 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 59263, sessionid =3D 0x13ae136197f0003, negotiated timeout =3D 10000 [junit4:junit4] 2> 82792 T1197 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@34054f name:ZooKeeperConnectio= n Watcher:127.0.0.1:59263/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 82792 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 82792 T1143 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 82793 T1143 oascc.SolrZkClient.makePath makePath: /liv= e_nodes/node1 [junit4:junit4] 2> 82795 T1191 oascc.ZkStateReader$3.process Updating liv= e nodes... (1) [junit4:junit4] 2> 82795 T1197 oascc.ZkStateReader$3.process Updating liv= e nodes... (1) [junit4:junit4] 2> 82795 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:59263/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1ac84ed [junit4:junit4] 2> 82796 T1198 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:59263 [junit4:junit4] 2> 82796 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 82796 T1198 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:59263, ini= tiating session [junit4:junit4] 2> 82796 T1186 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:35657 [junit4:junit4] 2> 82797 T1186 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:35657 [junit4:junit4] 2> 82797 T1188 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136197f0004 with negotiated timeout 10000 for client /= 127.0.0.1:35657 [junit4:junit4] 2> 82797 T1198 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 59263, sessionid =3D 0x13ae136197f0004, negotiated timeout =3D 10000 [junit4:junit4] 2> 82797 T1199 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1ac84ed name:ZooKeeperConnecti= on Watcher:127.0.0.1:59263/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 82798 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 82798 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/election [junit4:junit4] 2> 82801 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0004 type:de= lete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/ov= erseer_elect/leader [junit4:junit4] 2> 82801 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/leader [junit4:junit4] 2> 82802 T1143 oasc.Overseer.start Overseer (id=3D8863076= 5159383044-127.0.0.1:59263_solr-n_0000000000) starting [junit4:junit4] 2> 82803 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0004 type:cr= eate cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 82804 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0004 type:cr= eate cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 82804 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0004 type:cr= eate cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 82805 T1201 oasc.OverseerCollectionProcessor.run Proce= ss current queue of collection messages [junit4:junit4] 2> 82805 T1200 oasc.Overseer$ClusterStateUpdater.run Star= ting to work on the main queue [junit4:junit4] 2> 82900 T1181 oasc.Overseer$ClusterStateUpdater.amILeade= r WARNING org.apache.zookeeper.KeeperException$SessionExpiredException: Ke= eperErrorCode =3D Session expired for /overseer_elect/leader [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper= .java:927) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:244) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getDat= a(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.amILeader(Overseer.java:192) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.run(Overseer.java:116) [junit4:junit4] 2> =09at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2>=20 [junit4:junit4] 2> 82901 T1181 oasc.Overseer$ClusterStateUpdater.amILeade= r According to ZK I (id=3D88630759825276933-127.0.0.1:50631_solr-n_00000000= 02) am no longer a leader. [junit4:junit4] 2> 83805 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 83806 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer/queue Error:KeeperErrorCode =3D NoNode for /solr/overseer/= queue [junit4:junit4] 2> 83808 T1200 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 83809 T1200 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://node1/solr/"} [junit4:junit4] 2> 83809 T1200 oasc.Overseer$ClusterStateUpdater.createCo= llection Create collection collection1 with numShards 1 [junit4:junit4] 2> 83809 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0004 type:cr= eate cxid:0x23 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer/queue-work Error:KeeperErrorCode =3D NoNode for /solr/ove= rseer/queue-work [junit4:junit4] 2> 83812 T1191 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 83812 T1197 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 84308 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 84312 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:de= lete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/collections/collection1/leaders Error:KeeperErrorCode =3D NoNode f= or /solr/collections/collection1/leaders [junit4:junit4] 2> 84313 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leaders/shard1 [junit4:junit4] 2> 84315 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 84316 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 84317 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 84318 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 84319 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0x2e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperE= rrorCode =3D NodeExists for /solr/collections/collection1/leader_elect/shar= d1/election [junit4:junit4] 2> 84815 T1200 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 84831 T1200 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://node1/solr/"} [junit4:junit4] 2> 84834 T1191 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 84834 T1197 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 84885 T1189 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136197f0004 [junit4:junit4] 2> 84886 T1199 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 84886 T1186 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:35657 which had sessionid 0x13ae136197f00= 04 [junit4:junit4] 2> 84886 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f= 0004 closed [junit4:junit4] 2> 85342 T1200 oasc.Overseer$ClusterStateUpdater.amILeade= r WARNING org.apache.zookeeper.KeeperException$SessionExpiredException: Ke= eperErrorCode =3D Session expired for /overseer_elect/leader [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper= .java:927) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:244) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getDat= a(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.amILeader(Overseer.java:192) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.run(Overseer.java:116) [junit4:junit4] 2> =09at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2>=20 [junit4:junit4] 2> 85342 T1200 oasc.Overseer$ClusterStateUpdater.amILeade= r According to ZK I (id=3D88630765159383044-127.0.0.1:59263_solr-n_00000000= 00) am no longer a leader. [junit4:junit4] 2> 85887 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0x34 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 85888 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 85890 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0x3b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperE= rrorCode =3D NodeExists for /solr/collections/collection1/leader_elect/shar= d1/election [junit4:junit4] 2> 85891 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:59263/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@fdbc9 [junit4:junit4] 2> 85892 T1202 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:59263 [junit4:junit4] 2> 85892 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 85893 T1202 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:59263, ini= tiating session [junit4:junit4] 2> 85893 T1186 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:35674 [junit4:junit4] 2> 85893 T1186 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:35674 [junit4:junit4] 2> 85894 T1188 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136197f0005 with negotiated timeout 10000 for client /= 127.0.0.1:35674 [junit4:junit4] 2> 85894 T1202 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 59263, sessionid =3D 0x13ae136197f0005, negotiated timeout =3D 10000 [junit4:junit4] 2> 85894 T1203 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@fdbc9 name:ZooKeeperConnection= Watcher:127.0.0.1:59263/solr got event WatchedEvent state:SyncConnected ty= pe:None path:null path:null type:None [junit4:junit4] 2> 85894 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 85895 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0005 type:de= lete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/ov= erseer_elect/leader [junit4:junit4] 2> 85896 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/leader [junit4:junit4] 2> 85897 T1143 oasc.Overseer.start Overseer (id=3D8863076= 5159383045-127.0.0.1:59263_solr-n_0000000002) starting [junit4:junit4] 2> 85897 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0005 type:cr= eate cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 85897 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0005 type:cr= eate cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 85898 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0005 type:cr= eate cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 85898 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0005 type:cr= eate cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 85899 T1205 oasc.OverseerCollectionProcessor.run Proce= ss current queue of collection messages [junit4:junit4] 2> 85899 T1204 oasc.Overseer$ClusterStateUpdater.run Star= ting to work on the main queue [junit4:junit4] 2> 85900 T1204 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 85901 T1204 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://node1/solr/"} [junit4:junit4] 2> 85902 T1191 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 85902 T1197 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 85954 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0x41 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 85955 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 85956 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0x48 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperE= rrorCode =3D NodeExists for /solr/collections/collection1/leader_elect/shar= d1/election [junit4:junit4] 2> 86404 T1204 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 86408 T1197 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leaders/shard1 [junit4:junit4] 2> 86409 T1189 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136197f0003 type:cr= eate cxid:0x54 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/collections/collection1/leaders Error:KeeperErrorCode =3D NoNode f= or /solr/collections/collection1/leaders [junit4:junit4] 2> 86409 T1197 oasc.LeaderElector$1.process WARNING org.= apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =3D NoNod= e for /collections/collection1/leaders/shard1 [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:102) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.= java:643) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$10.exe= cute(SolrZkClient.java:399) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.makePa= th(SolrZkClient.java:396) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.makePa= th(SolrZkClient.java:353) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.makePa= th(SolrZkClient.java:340) [junit4:junit4] 2> =09at org.apache.solr.cloud.ShardLeaderElectionContext= Base.runLeaderProcess(ElectionContext.java:95) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.runIamLeader= Process(LeaderElector.java:155) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.checkIfIamLe= ader(LeaderElector.java:99) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.access$000(L= eaderElector.java:55) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector$1.process(Le= aderElector.java:128) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.proc= essEvent(ClientCnxn.java:526) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(= ClientCnxn.java:502) [junit4:junit4] 2>=20 [junit4:junit4] 2> 86411 T1197 oasc.LeaderElector$1.process WARNING org.= apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =3D NoNod= e for /collections/collection1/leader_elect/shard1/election [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:102) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getChildren(ZooKe= eper.java:1249) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$6.exec= ute(SolrZkClient.java:227) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$6.exec= ute(SolrZkClient.java:224) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getChi= ldren(SolrZkClient.java:224) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.checkIfIamLe= ader(LeaderElector.java:83) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.access$000(L= eaderElector.java:55) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector$1.process(Le= aderElector.java:128) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.proc= essEvent(ClientCnxn.java:526) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(= ClientCnxn.java:502) [junit4:junit4] 2>=20 [junit4:junit4] 2> 86411 T1197 oasc.LeaderElector$1.process WARNING org.= apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =3D NoNod= e for /collections/collection1/leader_elect/shard1/election [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:102) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getChildren(ZooKe= eper.java:1249) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$6.exec= ute(SolrZkClient.java:227) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$6.exec= ute(SolrZkClient.java:224) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getChi= ldren(SolrZkClient.java:224) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.checkIfIamLe= ader(LeaderElector.java:83) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.access$000(L= eaderElector.java:55) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector$1.process(Le= aderElector.java:128) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.proc= essEvent(ClientCnxn.java:526) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(= ClientCnxn.java:502) [junit4:junit4] 2>=20 [junit4:junit4] 2> 86412 T1191 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 86412 T1197 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 86913 T1191 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 86913 T1189 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136197f0003 [junit4:junit4] 2> 86913 T1197 oascc.ZkStateReader$3.process Updating liv= e nodes... (0) [junit4:junit4] 2> 86913 T1197 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (0) [junit4:junit4] 2> 86914 T1186 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:35656 which had sessionid 0x13ae136197f00= 03 [junit4:junit4] 2> 86914 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f= 0003 closed [junit4:junit4] 2> 86914 T1189 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136197f0005 [junit4:junit4] 2> 86914 T1191 oascc.ZkStateReader$3.process Updating liv= e nodes... (0) [junit4:junit4] 2> 86915 T1203 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 86915 T1186 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:35674 which had sessionid 0x13ae136197f00= 05 [junit4:junit4] 2> 86915 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f= 0005 closed [junit4:junit4] 2> 86915 T1189 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136197f0000 [junit4:junit4] 2> 86916 T1191 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 86916 T1186 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:35653 which had sessionid 0x13ae136197f00= 00 [junit4:junit4] 2> 86916 T1143 oaz.ZooKeeper.close Session: 0x13ae136197f= 0000 closed [junit4:junit4] 2> 86916 T1188 oazs.SyncRequestProcessor.run SyncRequestP= rocessor exited! [junit4:junit4] 2> 86916 T1189 oazs.PrepRequestProcessor.run PrepRequestP= rocessor exited loop! [junit4:junit4] 2> 86916 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 86918 T1186 oazs.NIOServerCnxn$Factory.run NIOServerCn= xn factory exited run method [junit4:junit4] 2> 86918 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 86918 T1143 oas.SolrTestCaseJ4.tearDown ###Ending test= OverseerFailure [junit4:junit4] 2> 86922 T1143 oas.SolrTestCaseJ4.setUp ###Starting testS= tateChange [junit4:junit4] 2> 86922 T1143 oasc.ZkTestServer.run STARTING ZK TEST SER= VER [junit4:junit4] 2> 86923 T1206 oazs.ZooKeeperServer.setTickTime tickTime = set to 1000 [junit4:junit4] 2> 86923 T1206 oazs.NIOServerCnxn$Factory. binding = to port 0.0.0.0/0.0.0.0:0 [junit4:junit4] 2> 86924 T1206 oazsp.FileSnap.deserialize Reading snapsho= t /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/t= est/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version= -2/snapshot.91 [junit4:junit4] 2> 86927 T1206 oazsp.FileTxnSnapLog.save Snapshotting: fd [junit4:junit4] 2> 87023 T1143 oasc.ZkTestServer.run start zk server on p= ort:35091 [junit4:junit4] 2> 87023 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:35091/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1ef3259 [junit4:junit4] 2> 87024 T1211 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:35091 [junit4:junit4] 2> 87024 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 87024 T1211 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:35091, ini= tiating session [junit4:junit4] 2> 87025 T1207 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:53265 [junit4:junit4] 2> 87025 T1207 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:53265 [junit4:junit4] 2> 87025 T1209 oazsp.FileTxnLog.append Creating new log f= ile: log.fe [junit4:junit4] 2> 87026 T1209 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362a220000 with negotiated timeout 10000 for client /= 127.0.0.1:53265 [junit4:junit4] 2> 87026 T1211 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 35091, sessionid =3D 0x13ae1362a220000, negotiated timeout =3D 10000 [junit4:junit4] 2> 87026 T1212 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1ef3259 name:ZooKeeperConnecti= on Watcher:127.0.0.1:35091/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 87027 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 87027 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:35091 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@55ca25 [junit4:junit4] 2> 87027 T1213 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:35091 [junit4:junit4] 2> 87027 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 87028 T1213 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:35091, ini= tiating session [junit4:junit4] 2> 87028 T1207 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:53266 [junit4:junit4] 2> 87028 T1207 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:53266 [junit4:junit4] 2> 87029 T1209 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362a220001 with negotiated timeout 10000 for client /= 127.0.0.1:53266 [junit4:junit4] 2> 87029 T1213 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 35091, sessionid =3D 0x13ae1362a220001, negotiated timeout =3D 10000 [junit4:junit4] 2> 87029 T1214 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@55ca25 name:ZooKeeperConnectio= n Watcher:127.0.0.1:35091 got event WatchedEvent state:SyncConnected type:N= one path:null path:null type:None [junit4:junit4] 2> 87029 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 87034 T1210 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362a220001 [junit4:junit4] 2> 87035 T1214 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 87035 T1207 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:53266 which had sessionid 0x13ae1362a2200= 01 [junit4:junit4] 2> 87035 T1143 oaz.ZooKeeper.close Session: 0x13ae1362a22= 0001 closed [junit4:junit4] 2> 87035 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:35091 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@862f9b [junit4:junit4] 2> 87036 T1215 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:35091 [junit4:junit4] 2> 87036 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 87036 T1215 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:35091, ini= tiating session [junit4:junit4] 2> 87036 T1207 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:53267 [junit4:junit4] 2> 87036 T1207 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:53267 [junit4:junit4] 2> 87037 T1209 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362a220002 with negotiated timeout 10000 for client /= 127.0.0.1:53267 [junit4:junit4] 2> 87037 T1215 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 35091, sessionid =3D 0x13ae1362a220002, negotiated timeout =3D 10000 [junit4:junit4] 2> 87037 T1216 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@862f9b name:ZooKeeperConnectio= n Watcher:127.0.0.1:35091 got event WatchedEvent state:SyncConnected type:N= one path:null path:null type:None [junit4:junit4] 2> 87037 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 87038 T1143 oascc.SolrZkClient.makePath makePath: /sol= r [junit4:junit4] 2> 87040 T1210 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362a220002 [junit4:junit4] 2> 87041 T1216 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 87041 T1207 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:53267 which had sessionid 0x13ae1362a2200= 02 [junit4:junit4] 2> 87041 T1143 oaz.ZooKeeper.close Session: 0x13ae1362a22= 0002 closed [junit4:junit4] 2> 87041 T1143 oascc.SolrZkClient.makePath makePath: /liv= e_nodes [junit4:junit4] 2> 87043 T1143 oascc.SolrZkClient.makePath makePath: /clu= sterstate.json [junit4:junit4] 2> 87044 T1143 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 87044 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:35091/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1498f23 [junit4:junit4] 2> 87045 T1217 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:35091 [junit4:junit4] 2> 87045 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 87045 T1217 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:35091, ini= tiating session [junit4:junit4] 2> 87045 T1207 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:53268 [junit4:junit4] 2> 87046 T1207 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:53268 [junit4:junit4] 2> 87046 T1209 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362a220003 with negotiated timeout 10000 for client /= 127.0.0.1:53268 [junit4:junit4] 2> 87046 T1217 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 35091, sessionid =3D 0x13ae1362a220003, negotiated timeout =3D 10000 [junit4:junit4] 2> 87046 T1218 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1498f23 name:ZooKeeperConnecti= on Watcher:127.0.0.1:35091/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 87047 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 87047 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/election [junit4:junit4] 2> 87049 T1210 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362a220003 type:de= lete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/ov= erseer_elect/leader [junit4:junit4] 2> 87050 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/leader [junit4:junit4] 2> 87051 T1143 oasc.Overseer.start Overseer (id=3D8863076= 5438500867-127.0.0.1:35091_solr-n_0000000000) starting [junit4:junit4] 2> 87051 T1210 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362a220003 type:cr= eate cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 87052 T1210 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362a220003 type:cr= eate cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 87052 T1210 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362a220003 type:cr= eate cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 87053 T1220 oasc.OverseerCollectionProcessor.run Proce= ss current queue of collection messages [junit4:junit4] 2> 87053 T1210 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362a220000 type:cr= eate cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 87054 T1210 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362a220000 type:cr= eate cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer/queue Error:KeeperErrorCode =3D NoNode for /solr/overseer/= queue [junit4:junit4] 2> 87054 T1219 oasc.Overseer$ClusterStateUpdater.run Star= ting to work on the main queue [junit4:junit4] 2> 87055 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 87055 T1219 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 87055 T1219 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3Dnull message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "roles":"", [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1/solr"} [junit4:junit4] 2> 87056 T1210 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362a220003 type:cr= eate cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer/queue-work Error:KeeperErrorCode =3D NoNode for /solr/ove= rseer/queue-work [junit4:junit4] 2> 87058 T1212 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (0) [junit4:junit4] 2> 87106 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 87414 T1204 oasc.Overseer$ClusterStateUpdater.amILeade= r WARNING org.apache.zookeeper.KeeperException$SessionExpiredException: Ke= eperErrorCode =3D Session expired for /overseer_elect/leader [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper= .java:927) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:244) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getDat= a(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.amILeader(Overseer.java:192) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.run(Overseer.java:116) [junit4:junit4] 2> =09at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2>=20 [junit4:junit4] 2> 87415 T1204 oasc.Overseer$ClusterStateUpdater.amILeade= r According to ZK I (id=3D88630765159383045-127.0.0.1:59263_solr-n_00000000= 02) am no longer a leader. [junit4:junit4] 2> 87560 T1219 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 87561 T1219 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3Dnull message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "roles":"", [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1/solr"} [junit4:junit4] 2> 87563 T1212 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (0) [junit4:junit4] 2> 87608 T1210 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362a220000 [junit4:junit4] 2> 87609 T1212 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 87609 T1207 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:53265 which had sessionid 0x13ae1362a2200= 00 [junit4:junit4] 2> 87609 T1143 oaz.ZooKeeper.close Session: 0x13ae1362a22= 0000 closed [junit4:junit4] 2> 87610 T1210 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362a220003 [junit4:junit4] 2> 87610 T1218 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 87610 T1207 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:53268 which had sessionid 0x13ae1362a2200= 03 [junit4:junit4] 2> 87610 T1143 oaz.ZooKeeper.close Session: 0x13ae1362a22= 0003 closed [junit4:junit4] 2> 87611 T1210 oazs.PrepRequestProcessor.run PrepRequestP= rocessor exited loop! [junit4:junit4] 2> 87611 T1209 oazs.SyncRequestProcessor.run SyncRequestP= rocessor exited! [junit4:junit4] 2> 87611 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 87612 T1207 oazs.NIOServerCnxn$Factory.run NIOServerCn= xn factory exited run method [junit4:junit4] 2> 87613 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 87613 T1143 oas.SolrTestCaseJ4.tearDown ###Ending test= StateChange [junit4:junit4] 2> 87617 T1143 oas.SolrTestCaseJ4.setUp ###Starting testD= oubleAssignment [junit4:junit4] 2> 87617 T1143 oasc.ZkTestServer.run STARTING ZK TEST SER= VER [junit4:junit4] 2> 87617 T1221 oazs.ZooKeeperServer.setTickTime tickTime = set to 1000 [junit4:junit4] 2> 87618 T1221 oazs.NIOServerCnxn$Factory. binding = to port 0.0.0.0/0.0.0.0:0 [junit4:junit4] 2> 87618 T1221 oazsp.FileSnap.deserialize Reading snapsho= t /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/t= est/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version= -2/snapshot.fd [junit4:junit4] 2> 87621 T1221 oazsp.FileTxnSnapLog.save Snapshotting: 12= b [junit4:junit4] 2> 87654 T1187 oazs.SessionTrackerImpl.run SessionTracker= Impl exited loop! [junit4:junit4] 2> 87654 T1208 oazs.SessionTrackerImpl.run SessionTracker= Impl exited loop! [junit4:junit4] 2> 87717 T1143 oasc.ZkTestServer.run start zk server on p= ort:57640 [junit4:junit4] 2> 87718 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:57640/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@b0ecde [junit4:junit4] 2> 87719 T1226 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:57640 [junit4:junit4] 2> 87719 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 87719 T1226 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:57640, ini= tiating session [junit4:junit4] 2> 87719 T1222 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:38742 [junit4:junit4] 2> 87719 T1222 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:38742 [junit4:junit4] 2> 87720 T1224 oazsp.FileTxnLog.append Creating new log f= ile: log.12c [junit4:junit4] 2> 87722 T1224 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362cd70000 with negotiated timeout 10000 for client /= 127.0.0.1:38742 [junit4:junit4] 2> 87722 T1226 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 57640, sessionid =3D 0x13ae1362cd70000, negotiated timeout =3D 10000 [junit4:junit4] 2> 87723 T1227 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@b0ecde name:ZooKeeperConnectio= n Watcher:127.0.0.1:57640/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 87723 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 87723 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:57640 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@1152a31 [junit4:junit4] 2> 87724 T1228 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:57640 [junit4:junit4] 2> 87724 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 87724 T1228 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:57640, ini= tiating session [junit4:junit4] 2> 87724 T1222 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:38743 [junit4:junit4] 2> 87725 T1222 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:38743 [junit4:junit4] 2> 87725 T1224 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362cd70001 with negotiated timeout 10000 for client /= 127.0.0.1:38743 [junit4:junit4] 2> 87725 T1228 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 57640, sessionid =3D 0x13ae1362cd70001, negotiated timeout =3D 10000 [junit4:junit4] 2> 87726 T1229 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1152a31 name:ZooKeeperConnecti= on Watcher:127.0.0.1:57640 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 87726 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 87731 T1225 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362cd70001 [junit4:junit4] 2> 87732 T1229 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 87732 T1222 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:38743 which had sessionid 0x13ae1362cd700= 01 [junit4:junit4] 2> 87732 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd7= 0001 closed [junit4:junit4] 2> 87732 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:57640 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@1ed5d72 [junit4:junit4] 2> 87733 T1230 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:57640 [junit4:junit4] 2> 87733 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 87733 T1230 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:57640, ini= tiating session [junit4:junit4] 2> 87733 T1222 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:38744 [junit4:junit4] 2> 87733 T1222 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:38744 [junit4:junit4] 2> 87734 T1224 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362cd70002 with negotiated timeout 10000 for client /= 127.0.0.1:38744 [junit4:junit4] 2> 87734 T1230 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 57640, sessionid =3D 0x13ae1362cd70002, negotiated timeout =3D 10000 [junit4:junit4] 2> 87734 T1231 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1ed5d72 name:ZooKeeperConnecti= on Watcher:127.0.0.1:57640 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 87734 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 87735 T1143 oascc.SolrZkClient.makePath makePath: /sol= r [junit4:junit4] 2> 87736 T1225 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362cd70002 [junit4:junit4] 2> 87736 T1231 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 87736 T1222 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:38744 which had sessionid 0x13ae1362cd700= 02 [junit4:junit4] 2> 87736 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd7= 0002 closed [junit4:junit4] 2> 87737 T1143 oascc.SolrZkClient.makePath makePath: /liv= e_nodes [junit4:junit4] 2> 87738 T1143 oascc.SolrZkClient.makePath makePath: /clu= sterstate.json [junit4:junit4] 2> 87739 T1143 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 87740 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:57640/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@17491f8 [junit4:junit4] 2> 87741 T1232 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:57640 [junit4:junit4] 2> 87741 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 87741 T1232 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:57640, ini= tiating session [junit4:junit4] 2> 87741 T1222 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:38745 [junit4:junit4] 2> 87741 T1222 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:38745 [junit4:junit4] 2> 87742 T1224 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362cd70003 with negotiated timeout 10000 for client /= 127.0.0.1:38745 [junit4:junit4] 2> 87742 T1232 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 57640, sessionid =3D 0x13ae1362cd70003, negotiated timeout =3D 10000 [junit4:junit4] 2> 87742 T1233 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@17491f8 name:ZooKeeperConnecti= on Watcher:127.0.0.1:57640/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 87742 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 87750 T1143 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 87750 T1143 oascc.SolrZkClient.makePath makePath: /liv= e_nodes/node1 [junit4:junit4] 2> 87753 T1227 oascc.ZkStateReader$3.process Updating liv= e nodes... (1) [junit4:junit4] 2> 87753 T1233 oascc.ZkStateReader$3.process Updating liv= e nodes... (1) [junit4:junit4] 2> 87753 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:57640/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1527903 [junit4:junit4] 2> 87754 T1234 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:57640 [junit4:junit4] 2> 87754 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 87754 T1234 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:57640, ini= tiating session [junit4:junit4] 2> 87754 T1222 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:38746 [junit4:junit4] 2> 87754 T1222 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:38746 [junit4:junit4] 2> 87755 T1224 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362cd70004 with negotiated timeout 10000 for client /= 127.0.0.1:38746 [junit4:junit4] 2> 87755 T1234 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 57640, sessionid =3D 0x13ae1362cd70004, negotiated timeout =3D 10000 [junit4:junit4] 2> 87755 T1235 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1527903 name:ZooKeeperConnecti= on Watcher:127.0.0.1:57640/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 87755 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 87756 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/election [junit4:junit4] 2> 87758 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70004 type:de= lete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/ov= erseer_elect/leader [junit4:junit4] 2> 87759 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/leader [junit4:junit4] 2> 87760 T1143 oasc.Overseer.start Overseer (id=3D8863076= 5483917316-127.0.0.1:57640_solr-n_0000000000) starting [junit4:junit4] 2> 87760 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70004 type:cr= eate cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 87761 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70004 type:cr= eate cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 87761 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70004 type:cr= eate cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 87762 T1237 oasc.OverseerCollectionProcessor.run Proce= ss current queue of collection messages [junit4:junit4] 2> 87762 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70003 type:cr= eate cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 87763 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70003 type:cr= eate cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer/queue Error:KeeperErrorCode =3D NoNode for /solr/overseer/= queue [junit4:junit4] 2> 87763 T1236 oasc.Overseer$ClusterStateUpdater.run Star= ting to work on the main queue [junit4:junit4] 2> 87914 T1197 oascc.ZkStateReader$2.process WARNING ZooK= eeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 87914 T1197 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 88064 T1219 oasc.Overseer$ClusterStateUpdater.amILeade= r WARNING org.apache.zookeeper.KeeperException$SessionExpiredException: Ke= eperErrorCode =3D Session expired for /overseer_elect/leader [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper= .java:927) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:244) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getDat= a(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.amILeader(Overseer.java:192) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.run(Overseer.java:116) [junit4:junit4] 2> =09at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2>=20 [junit4:junit4] 2> 88065 T1219 oasc.Overseer$ClusterStateUpdater.amILeade= r According to ZK I (id=3D88630765438500867-127.0.0.1:35091_solr-n_00000000= 00) am no longer a leader. [junit4:junit4] 2> 88265 T1236 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 88266 T1236 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://node1/solr/"} [junit4:junit4] 2> 88266 T1236 oasc.Overseer$ClusterStateUpdater.createCo= llection Create collection collection1 with numShards 1 [junit4:junit4] 2> 88266 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70004 type:cr= eate cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer/queue-work Error:KeeperErrorCode =3D NoNode for /solr/ove= rseer/queue-work [junit4:junit4] 2> 88269 T1227 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 88269 T1233 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 88764 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 88768 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70003 type:de= lete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/collections/collection1/leaders Error:KeeperErrorCode =3D NoNode f= or /solr/collections/collection1/leaders [junit4:junit4] 2> 88769 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leaders/shard1 [junit4:junit4] 2> 88771 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70003 type:cr= eate cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 88772 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 88773 T1233 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 88773 T1225 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362cd70003 [junit4:junit4] 2> 88773 T1227 oascc.ZkStateReader$3.process Updating liv= e nodes... (0) [junit4:junit4] 2> 88773 T1227 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (0) [junit4:junit4] 2> 88773 T1222 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:38745 which had sessionid 0x13ae1362cd700= 03 [junit4:junit4] 2> 88773 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd7= 0003 closed [junit4:junit4] 2> 88774 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:57640/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1d70b1d [junit4:junit4] 2> 88775 T1238 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:57640 [junit4:junit4] 2> 88775 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 88775 T1238 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:57640, ini= tiating session [junit4:junit4] 2> 88775 T1222 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:38747 [junit4:junit4] 2> 88776 T1222 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:38747 [junit4:junit4] 2> 88776 T1238 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 57640, sessionid =3D 0x13ae1362cd70005, negotiated timeout =3D 10000 [junit4:junit4] 2> 88776 T1224 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae1362cd70005 with negotiated timeout 10000 for client /= 127.0.0.1:38747 [junit4:junit4] 2> 88777 T1239 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1d70b1d name:ZooKeeperConnecti= on Watcher:127.0.0.1:57640/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 88777 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 88777 T1143 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 88778 T1143 oascc.SolrZkClient.makePath makePath: /liv= e_nodes/node1 [junit4:junit4] 2> 88780 T1227 oascc.ZkStateReader$3.process Updating liv= e nodes... (1) [junit4:junit4] 2> 88780 T1239 oascc.ZkStateReader$3.process Updating liv= e nodes... (1) [junit4:junit4] 2> 88780 T1227 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 88780 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70005 type:cr= eate cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 88781 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 88782 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70005 type:cr= eate cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/collections/collection1/leader_elect/shard1/election Error:KeeperE= rrorCode =3D NodeExists for /solr/collections/collection1/leader_elect/shar= d1/election [junit4:junit4] 2> 88783 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70005 type:de= lete cxid:0x14 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/collections/collection1/leaders/shard1 Error:KeeperErrorCode =3D N= oNode for /solr/collections/collection1/leaders/shard1 [junit4:junit4] 2> 88783 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leaders/shard1 [junit4:junit4] 2> 88785 T1225 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae1362cd70005 type:cr= eate cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89272 T1236 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 89274 T1236 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://node1/solr/"} [junit4:junit4] 2> 89279 T1227 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 89279 T1239 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 89280 T1143 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 89282 T1225 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362cd70004 [junit4:junit4] 2> 89283 T1235 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 89282 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd7= 0004 closed [junit4:junit4] 2> 89283 T1222 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:38746 which had sessionid 0x13ae1362cd700= 04 [junit4:junit4] 2> 89284 T1239 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 89285 T1225 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362cd70005 [junit4:junit4] 2> 89285 T1222 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:38747 which had sessionid 0x13ae1362cd700= 05 [junit4:junit4] 2> 89285 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd7= 0005 closed [junit4:junit4] 2> 89286 T1227 oascc.ZkStateReader$3.process Updating liv= e nodes... (0) [junit4:junit4] 2> 89286 T1225 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae1362cd70000 [junit4:junit4] 2> 89286 T1227 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (0) [junit4:junit4] 2> 89286 T1222 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:38742 which had sessionid 0x13ae1362cd700= 00 [junit4:junit4] 2> 89286 T1143 oaz.ZooKeeper.close Session: 0x13ae1362cd7= 0000 closed [junit4:junit4] 2> 89287 T1225 oazs.PrepRequestProcessor.run PrepRequestP= rocessor exited loop! [junit4:junit4] 2> 89287 T1224 oazs.SyncRequestProcessor.run SyncRequestP= rocessor exited! [junit4:junit4] 2> 89287 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 89289 T1222 oazs.NIOServerCnxn$Factory.run NIOServerCn= xn factory exited run method [junit4:junit4] 2> 89289 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 89290 T1143 oas.SolrTestCaseJ4.tearDown ###Ending test= DoubleAssignment [junit4:junit4] 2> 89296 T1143 oas.SolrTestCaseJ4.setUp ###Starting testR= eplay [junit4:junit4] 2> 89296 T1143 oasc.ZkTestServer.run STARTING ZK TEST SER= VER [junit4:junit4] 2> 89297 T1240 oazs.ZooKeeperServer.setTickTime tickTime = set to 1000 [junit4:junit4] 2> 89297 T1240 oazs.NIOServerCnxn$Factory. binding = to port 0.0.0.0/0.0.0.0:0 [junit4:junit4] 2> 89300 T1240 oazsp.FileSnap.deserialize Reading snapsho= t /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/t= est/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version= -2/snapshot.12b [junit4:junit4] 2> 89303 T1240 oazsp.FileTxnSnapLog.save Snapshotting: 17= 6 [junit4:junit4] 2> 89397 T1143 oasc.ZkTestServer.run start zk server on p= ort:42634 [junit4:junit4] 2> 89398 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:42634/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1644431 [junit4:junit4] 2> 89399 T1245 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:42634 [junit4:junit4] 2> 89399 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 89399 T1241 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:48715 [junit4:junit4] 2> 89399 T1245 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:42634, ini= tiating session [junit4:junit4] 2> 89400 T1241 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:48715 [junit4:junit4] 2> 89400 T1243 oazsp.FileTxnLog.append Creating new log f= ile: log.177 [junit4:junit4] 2> 89413 T1245 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 42634, sessionid =3D 0x13ae136336a0000, negotiated timeout =3D 10000 [junit4:junit4] 2> 89413 T1243 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136336a0000 with negotiated timeout 10000 for client /= 127.0.0.1:48715 [junit4:junit4] 2> 89413 T1246 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1644431 name:ZooKeeperConnecti= on Watcher:127.0.0.1:42634/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 89413 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 89414 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:42634 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@12b0019 [junit4:junit4] 2> 89415 T1247 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:42634 [junit4:junit4] 2> 89415 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 89415 T1247 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:42634, ini= tiating session [junit4:junit4] 2> 89415 T1241 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:48716 [junit4:junit4] 2> 89415 T1241 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:48716 [junit4:junit4] 2> 89426 T1247 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 42634, sessionid =3D 0x13ae136336a0001, negotiated timeout =3D 10000 [junit4:junit4] 2> 89426 T1243 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136336a0001 with negotiated timeout 10000 for client /= 127.0.0.1:48716 [junit4:junit4] 2> 89426 T1248 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@12b0019 name:ZooKeeperConnecti= on Watcher:127.0.0.1:42634 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 89426 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 89434 T1244 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136336a0001 [junit4:junit4] 2> 89435 T1248 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 89435 T1241 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:48716 which had sessionid 0x13ae136336a00= 01 [junit4:junit4] 2> 89435 T1143 oaz.ZooKeeper.close Session: 0x13ae136336a= 0001 closed [junit4:junit4] 2> 89450 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:42634 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@162e705 [junit4:junit4] 2> 89451 T1249 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:42634 [junit4:junit4] 2> 89451 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 89452 T1249 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:42634, ini= tiating session [junit4:junit4] 2> 89452 T1241 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:48717 [junit4:junit4] 2> 89452 T1241 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:48717 [junit4:junit4] 2> 89453 T1249 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 42634, sessionid =3D 0x13ae136336a0002, negotiated timeout =3D 10000 [junit4:junit4] 2> 89453 T1243 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136336a0002 with negotiated timeout 10000 for client /= 127.0.0.1:48717 [junit4:junit4] 2> 89454 T1250 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@162e705 name:ZooKeeperConnecti= on Watcher:127.0.0.1:42634 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 89454 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 89454 T1143 oascc.SolrZkClient.makePath makePath: /sol= r [junit4:junit4] 2> 89456 T1244 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136336a0002 [junit4:junit4] 2> 89456 T1250 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 89456 T1241 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:48717 which had sessionid 0x13ae136336a00= 02 [junit4:junit4] 2> 89456 T1143 oaz.ZooKeeper.close Session: 0x13ae136336a= 0002 closed [junit4:junit4] 2> 89457 T1143 oascc.SolrZkClient.makePath makePath: /liv= e_nodes [junit4:junit4] 2> 89458 T1143 oascc.SolrZkClient.makePath makePath: /clu= sterstate.json [junit4:junit4] 2> 89459 T1143 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 89460 T1244 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136336a0000 type:cr= eate cxid:0xc zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer/queue-work Error:KeeperErrorCode =3D NoNode for /solr/over= seer/queue-work [junit4:junit4] 2> 89462 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:42634/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@e5b7ca [junit4:junit4] 2> 89462 T1251 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:42634 [junit4:junit4] 2> 89462 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 89463 T1251 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:42634, ini= tiating session [junit4:junit4] 2> 89463 T1241 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:48718 [junit4:junit4] 2> 89463 T1241 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:48718 [junit4:junit4] 2> 89464 T1243 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae136336a0003 with negotiated timeout 10000 for client /= 127.0.0.1:48718 [junit4:junit4] 2> 89464 T1251 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 42634, sessionid =3D 0x13ae136336a0003, negotiated timeout =3D 10000 [junit4:junit4] 2> 89464 T1252 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@e5b7ca name:ZooKeeperConnectio= n Watcher:127.0.0.1:42634/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 89464 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 89465 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/election [junit4:junit4] 2> 89466 T1244 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136336a0003 type:de= lete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/ov= erseer_elect/leader [junit4:junit4] 2> 89467 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/leader [junit4:junit4] 2> 89468 T1143 oasc.Overseer.start Overseer (id=3D8863076= 5594214403-127.0.0.1:42634_solr-n_0000000000) starting [junit4:junit4] 2> 89468 T1244 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136336a0003 type:cr= eate cxid:0xe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89469 T1244 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136336a0003 type:cr= eate cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89469 T1244 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136336a0003 type:cr= eate cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89469 T1244 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136336a0003 type:cr= eate cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89471 T1254 oasc.OverseerCollectionProcessor.run Proce= ss current queue of collection messages [junit4:junit4] 2> 89471 T1244 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136336a0000 type:cr= eate cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89471 T1244 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae136336a0000 type:cr= eate cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer/queue Error:KeeperErrorCode =3D NoNode for /solr/overseer= /queue [junit4:junit4] 2> 89473 T1253 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 89473 T1253 oasc.Overseer$ClusterStateUpdater.run Repl= aying operations from work queue. [junit4:junit4] 2> 89473 T1253 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3Dnull message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "shard":"s1", [junit4:junit4] 2> =09 "roles":"", [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1/solr"} [junit4:junit4] 2> 89474 T1246 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (0) [junit4:junit4] 2> 89475 T1253 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3Dnull message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "shard":"s1", [junit4:junit4] 2> =09 "roles":"", [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"core2", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1/solr"} [junit4:junit4] 2> 89476 T1246 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (0) [junit4:junit4] 2> 89477 T1253 oasc.Overseer$ClusterStateUpdater.run Star= ting to work on the main queue [junit4:junit4] 2> 89477 T1253 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 89477 T1253 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3Dnull message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "shard":"s1", [junit4:junit4] 2> =09 "roles":"", [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"core3", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"node1", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1/solr"} [junit4:junit4] 2> 89479 T1246 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (0) [junit4:junit4] 2> 89573 T1244 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136336a0003 [junit4:junit4] 2> 89574 T1252 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 89574 T1241 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:48718 which had sessionid 0x13ae136336a00= 03 [junit4:junit4] 2> 89574 T1143 oaz.ZooKeeper.close Session: 0x13ae136336a= 0003 closed [junit4:junit4] 2> 89574 T1244 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae136336a0000 [junit4:junit4] 2> 89575 T1246 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 89575 T1241 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:48715 which had sessionid 0x13ae136336a00= 00 [junit4:junit4] 2> 89575 T1143 oaz.ZooKeeper.close Session: 0x13ae136336a= 0000 closed [junit4:junit4] 2> 89575 T1244 oazs.PrepRequestProcessor.run PrepRequestP= rocessor exited loop! [junit4:junit4] 2> 89575 T1243 oazs.SyncRequestProcessor.run SyncRequestP= rocessor exited! [junit4:junit4] 2> 89576 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 89577 T1241 oazs.NIOServerCnxn$Factory.run NIOServerCn= xn factory exited run method [junit4:junit4] 2> 89577 T1143 oazs.FinalRequestProcessor.shutdown shutdo= wn of request processor complete [junit4:junit4] 2> 89578 T1143 oas.SolrTestCaseJ4.tearDown ###Ending test= Replay [junit4:junit4] 2> 89582 T1143 oas.SolrTestCaseJ4.setUp ###Starting testS= hardAssignment [junit4:junit4] 2> 89582 T1143 oasc.ZkTestServer.run STARTING ZK TEST SER= VER [junit4:junit4] 2> 89583 T1255 oazs.ZooKeeperServer.setTickTime tickTime = set to 1000 [junit4:junit4] 2> 89583 T1255 oazs.NIOServerCnxn$Factory. binding = to port 0.0.0.0/0.0.0.0:0 [junit4:junit4] 2> 89583 T1255 oazsp.FileSnap.deserialize Reading snapsho= t /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/t= est/J1/./solrtest-OverseerTest-1352398067346/zookeeper/server1/data/version= -2/snapshot.176 [junit4:junit4] 2> 89588 T1255 oazsp.FileTxnSnapLog.save Snapshotting: 1a= d [junit4:junit4] 2> 89654 T1223 oazs.SessionTrackerImpl.run SessionTracker= Impl exited loop! [junit4:junit4] 2> 89654 T1242 oazs.SessionTrackerImpl.run SessionTracker= Impl exited loop! [junit4:junit4] 2> 89683 T1143 oasc.ZkTestServer.run start zk server on p= ort:51735 [junit4:junit4] 2> 89683 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:51735 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@10307f7 [junit4:junit4] 2> 89684 T1260 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:51735 [junit4:junit4] 2> 89684 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 89684 T1260 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:51735, ini= tiating session [junit4:junit4] 2> 89684 T1256 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:49784 [junit4:junit4] 2> 89685 T1256 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:49784 [junit4:junit4] 2> 89685 T1258 oazsp.FileTxnLog.append Creating new log f= ile: log.1ae [junit4:junit4] 2> 89686 T1258 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae13634860000 with negotiated timeout 10000 for client /= 127.0.0.1:49784 [junit4:junit4] 2> 89686 T1260 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 51735, sessionid =3D 0x13ae13634860000, negotiated timeout =3D 10000 [junit4:junit4] 2> 89686 T1261 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@10307f7 name:ZooKeeperConnecti= on Watcher:127.0.0.1:51735 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 89687 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 89691 T1259 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae13634860000 [junit4:junit4] 2> 89692 T1256 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:49784 which had sessionid 0x13ae136348600= 00 [junit4:junit4] 2> 89692 T1143 oaz.ZooKeeper.close Session: 0x13ae1363486= 0000 closed [junit4:junit4] 2> 89692 T1261 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 89692 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:51735 sessionTimeout=3D10000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@1d66a30 [junit4:junit4] 2> 89693 T1262 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:51735 [junit4:junit4] 2> 89693 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 89693 T1256 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:49785 [junit4:junit4] 2> 89693 T1262 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:51735, ini= tiating session [junit4:junit4] 2> 89693 T1256 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:49785 [junit4:junit4] 2> 89694 T1258 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae13634860001 with negotiated timeout 10000 for client /= 127.0.0.1:49785 [junit4:junit4] 2> 89694 T1262 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 51735, sessionid =3D 0x13ae13634860001, negotiated timeout =3D 10000 [junit4:junit4] 2> 89694 T1263 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1d66a30 name:ZooKeeperConnecti= on Watcher:127.0.0.1:51735 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 89694 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 89695 T1143 oascc.SolrZkClient.makePath makePath: /sol= r [junit4:junit4] 2> 89696 T1259 oazs.PrepRequestProcessor.pRequest Process= ed session termination for sessionid: 0x13ae13634860001 [junit4:junit4] 2> 89696 T1256 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:49785 which had sessionid 0x13ae136348600= 01 [junit4:junit4] 2> 89696 T1143 oaz.ZooKeeper.close Session: 0x13ae1363486= 0001 closed [junit4:junit4] 2> 89697 T1263 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 89697 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:51735/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@7a8dfc [junit4:junit4] 2> 89697 T1264 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:51735 [junit4:junit4] 2> 89697 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 89698 T1256 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:49786 [junit4:junit4] 2> 89698 T1264 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:51735, ini= tiating session [junit4:junit4] 2> 89698 T1256 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:49786 [junit4:junit4] 2> 89700 T1258 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae13634860002 with negotiated timeout 10000 for client /= 127.0.0.1:49786 [junit4:junit4] 2> 89700 T1264 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 51735, sessionid =3D 0x13ae13634860002, negotiated timeout =3D 10000 [junit4:junit4] 2> 89700 T1265 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@7a8dfc name:ZooKeeperConnectio= n Watcher:127.0.0.1:51735/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 89700 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 89700 T1143 oascc.SolrZkClient.makePath makePath: /liv= e_nodes [junit4:junit4] 2> 89702 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:51735/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@5bbeb7 [junit4:junit4] 2> 89702 T1266 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:51735 [junit4:junit4] 2> 89702 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 89703 T1266 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:51735, ini= tiating session [junit4:junit4] 2> 89703 T1256 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:49787 [junit4:junit4] 2> 89703 T1256 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:49787 [junit4:junit4] 2> 89704 T1258 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae13634860003 with negotiated timeout 10000 for client /= 127.0.0.1:49787 [junit4:junit4] 2> 89704 T1266 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 51735, sessionid =3D 0x13ae13634860003, negotiated timeout =3D 10000 [junit4:junit4] 2> 89704 T1267 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@5bbeb7 name:ZooKeeperConnectio= n Watcher:127.0.0.1:51735/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 89704 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 89705 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/election [junit4:junit4] 2> 89707 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860003 type:de= lete cxid:0x9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer_elect/leader Error:KeeperErrorCode =3D NoNode for /solr/ov= erseer_elect/leader [junit4:junit4] 2> 89707 T1143 oascc.SolrZkClient.makePath makePath: /ove= rseer_elect/leader [junit4:junit4] 2> 89708 T1143 oasc.Overseer.start Overseer (id=3D8863076= 5612826627-127.0.0.1:51735_solr-n_0000000000) starting [junit4:junit4] 2> 89709 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860003 type:cr= eate cxid:0xf zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89709 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860003 type:cr= eate cxid:0x10 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89710 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860003 type:cr= eate cxid:0x11 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89711 T1269 oasc.OverseerCollectionProcessor.run Proce= ss current queue of collection messages [junit4:junit4] 2> 89711 T1143 oascc.SolrZkClient.makePath makePath: /clu= sterstate.json [junit4:junit4] 2> 89712 T1268 oasc.Overseer$ClusterStateUpdater.run Star= ting to work on the main queue [junit4:junit4] 2> 89713 T1143 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 89714 T1143 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:51735/solr sessionTimeout=3D10000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@9161f0 [junit4:junit4] 2> 89714 T1270 oaz.ClientCnxn$SendThread.startConnect Ope= ning socket connection to server /127.0.0.1:51735 [junit4:junit4] 2> 89714 T1143 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 89715 T1270 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:51735, ini= tiating session [junit4:junit4] 2> 89715 T1256 oazs.NIOServerCnxn$Factory.run Accepted so= cket connection from /127.0.0.1:49788 [junit4:junit4] 2> 89715 T1256 oazs.NIOServerCnxn.readConnectRequest Clie= nt attempting to establish new session at /127.0.0.1:49788 [junit4:junit4] 2> 89716 T1258 oazs.NIOServerCnxn.finishSessionInit Estab= lished session 0x13ae13634860004 with negotiated timeout 10000 for client /= 127.0.0.1:49788 [junit4:junit4] 2> 89716 T1270 oaz.ClientCnxn$SendThread.readConnectResul= t Session establishment complete on server localhost.localdomain/127.0.0.1:= 51735, sessionid =3D 0x13ae13634860004, negotiated timeout =3D 10000 [junit4:junit4] 2> 89716 T1271 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@9161f0 name:ZooKeeperConnectio= n Watcher:127.0.0.1:51735/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 89716 T1143 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 89717 T1143 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 89718 T1143 oascc.SolrZkClient.makePath makePath: /liv= e_nodes/127.0.0.1 [junit4:junit4] 2> 89721 T1271 oascc.ZkStateReader$3.process Updating liv= e nodes... (1) [junit4:junit4] 2> 89721 T1265 oascc.ZkStateReader$3.process Updating liv= e nodes... (1) [junit4:junit4] 2> 89721 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860004 type:cr= eate cxid:0xa zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 89721 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860004 type:cr= eate cxid:0xb zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pat= h:/solr/overseer/queue Error:KeeperErrorCode =3D NoNode for /solr/overseer/= queue [junit4:junit4] 2> 89773 T1233 oascc.ZkStateReader$2.process WARNING ZooK= eeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 89773 T1233 oascc.ZkStateReader$3.process WARNING ZooK= eeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 89774 T1233 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 89980 T1253 oasc.Overseer$ClusterStateUpdater.amILeade= r WARNING org.apache.zookeeper.KeeperException$SessionExpiredException: Ke= eperErrorCode =3D Session expired for /overseer_elect/leader [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper= .java:927) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:244) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$7.exec= ute(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.getDat= a(SolrZkClient.java:241) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.amILeader(Overseer.java:192) [junit4:junit4] 2> =09at org.apache.solr.cloud.Overseer$ClusterStateUpdat= er.run(Overseer.java:116) [junit4:junit4] 2> =09at java.lang.Thread.run(Thread.java:662) [junit4:junit4] 2>=20 [junit4:junit4] 2> 89980 T1253 oasc.Overseer$ClusterStateUpdater.amILeade= r According to ZK I (id=3D88630765594214403-127.0.0.1:42634_solr-n_00000000= 00) am no longer a leader. [junit4:junit4] 2> 90214 T1268 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 90214 T1268 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"core1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1/solr/"} [junit4:junit4] 2> 90215 T1268 oasc.Overseer$ClusterStateUpdater.createCo= llection Create collection collection1 with numShards 3 [junit4:junit4] 2> 90215 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860003 type:cr= eate cxid:0x20 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer/queue-work Error:KeeperErrorCode =3D NoNode for /solr/ove= rseer/queue-work [junit4:junit4] 2> 90218 T1271 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 90218 T1265 oascc.ZkStateReader$2.process A cluster st= ate change has occurred - updating... (1) [junit4:junit4] 2> 90222 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 90227 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860004 type:de= lete cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/collections/collection1/leaders Error:KeeperErrorCode =3D NoNode f= or /solr/collections/collection1/leaders [junit4:junit4] 2> 90227 T1143 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leaders/shard1 [junit4:junit4] 2> 90229 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860004 type:cr= eate cxid:0x25 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 90230 T1259 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13ae13634860004 type:cr= eate cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Pa= th:/solr/overseer Error:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 90282 T1236 oasc.Overseer$ClusterStateUpdater.run WARN= ING Overseer cannot talk to ZK [junit4:junit4] 2> 90285 T1239 oascc.ZkStateReader$2.process WARNING ZooK= eeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 90285 T1239 oasc [...truncated too long message...] Keeper.java:815) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$3.exec= ute(SolrZkClient.java:176) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$3.exec= ute(SolrZkClient.java:173) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.exists= (SolrZkClient.java:173) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.makePa= th(SolrZkClient.java:384) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.makePa= th(SolrZkClient.java:353) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.makePa= th(SolrZkClient.java:340) [junit4:junit4] 2> =09at org.apache.solr.cloud.ShardLeaderElectionContext= Base.runLeaderProcess(ElectionContext.java:95) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.runIamLeader= Process(LeaderElector.java:155) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.checkIfIamLe= ader(LeaderElector.java:99) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.access$000(L= eaderElector.java:55) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector$1.process(Le= aderElector.java:128) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.proc= essEvent(ClientCnxn.java:526) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(= ClientCnxn.java:502) [junit4:junit4] 2>=20 [junit4:junit4] 2> 103215 T1414 oascc.ZkStateReader$3.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103216 T1414 oascc.ZkStateReader$2.process A cluster s= tate change has occurred - updating... (18) [junit4:junit4] 2> 103216 T1414 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103216 T1414 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103223 T1416 oascc.ZkStateReader$3.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103223 T1416 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103228 T1418 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103228 T1418 oascc.ZkStateReader$3.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103229 T1418 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103252 T1422 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103253 T1422 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103271 T1424 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103272 T1424 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103293 T1426 oasc.LeaderElector$1.process WARNING org= .apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = =3D Session expired for /overseer/queue/qn- [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:118) [junit4:junit4] 2> =09at org.apache.zookeeper.KeeperException.create(Keep= erException.java:42) [junit4:junit4] 2> =09at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.= java:643) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$5.exec= ute(SolrZkClient.java:210) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient$5.exec= ute(SolrZkClient.java:207) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkCmdExecutor.retry= Operation(ZkCmdExecutor.java:63) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.SolrZkClient.create= (SolrZkClient.java:207) [junit4:junit4] 2> =09at org.apache.solr.cloud.DistributedQueue.offer(Dis= tributedQueue.java:226) [junit4:junit4] 2> =09at org.apache.solr.cloud.ShardLeaderElectionContext= Base.runLeaderProcess(ElectionContext.java:104) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.runIamLeader= Process(LeaderElector.java:155) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.checkIfIamLe= ader(LeaderElector.java:99) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector.access$000(L= eaderElector.java:55) [junit4:junit4] 2> =09at org.apache.solr.cloud.LeaderElector$1.process(Le= aderElector.java:128) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.proc= essEvent(ClientCnxn.java:526) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(= ClientCnxn.java:502) [junit4:junit4] 2>=20 [junit4:junit4] 2> 103293 T1426 oascc.ZkStateReader$2.process A cluster s= tate change has occurred - updating... (13) [junit4:junit4] 2> 103293 T1426 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103293 T1426 oascc.ZkStateReader$3.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103293 T1426 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103300 T1430 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103300 T1430 oascc.ZkStateReader$3.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103301 T1430 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103304 T1432 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103305 T1432 oascc.ZkStateReader$3.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103305 T1432 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103315 T1436 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103315 T1436 oascc.ZkStateReader$3.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103316 T1436 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103332 T1438 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103333 T1438 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103335 T1440 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103335 T1440 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103340 T1442 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103341 T1442 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103343 T1444 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103344 T1444 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 103345 T1446 oascc.ZkStateReader$2.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103345 T1446 oascc.ZkStateReader$3.process WARNING Zoo= Keeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 103345 T1446 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 105350 T1143 oas.SolrTestCaseJ4.deleteCore ###deleteCo= re [junit4:junit4] 2> NOTE: test params are: codec=3DAsserting, sim=3DDefaul= tSimilarity, locale=3Dnl_NL, timezone=3DAmerica/Aruba [junit4:junit4] 2> NOTE: Linux 3.2.0-32-generic i386/Sun Microsystems Inc= . 1.6.0_37 (32-bit)/cpus=3D8,threads=3D10,free=3D210099408,total=3D30290739= 2 [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestSort, TestCSVRes= ponseWriter, TestFunctionQuery, ClusterStateTest, ResponseLogComponentTest,= LegacyHTMLStripCharFilterTest, FieldMutatingUpdateProcessorTest, TestSolrD= eletionPolicy1, OutputWriterTest, MultiTermTest, SyncSliceTest, TestDistrib= utedSearch, ShowFileRequestHandlerTest, TestQuerySenderListener, RecoveryZk= Test, IndexSchemaRuntimeFieldTest, XsltUpdateRequestHandlerTest, TestPhrase= Suggestions, PeerSyncTest, SolrInfoMBeanTest, TestDistributedGrouping, Test= PluginEnable, DistributedQueryElevationComponentTest, FullSolrCloudDistribC= mdsTest, TestReversedWildcardFilterFactory, DebugComponentTest, TestFolding= MultitermQuery, TestRecovery, TestWriterPerf, TestQueryUtils, ConvertedLega= cyTest, TestBinaryField, XmlUpdateRequestHandlerTest, TestSolrDeletionPolic= y2, TestSurroundQueryParser, DocumentBuilderTest, DirectUpdateHandlerTest, = DocumentAnalysisRequestHandlerTest, StatsComponentTest, TestStressVersions,= FileUtilsTest, BasicZkTest, TestValueSourceCache, TimeZoneUtilsTest, PrimU= tilsTest, TestDocSet, TestSolrJ, SuggesterTest, QueryEqualityTest, ZkNodePr= opsTest, SignatureUpdateProcessorFactoryTest, DirectSolrSpellCheckerTest, O= verseerTest] [junit4:junit4] Completed on J1 in 105.37s, 8 tests, 1 failure <<< FAILURES= ! [...truncated 375 lines...] BUILD FAILED /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:335: The follo= wing error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The follow= ing error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:178: The = following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:= 61: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:= 1163: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:= 827: There were test failures: 233 suites, 970 tests, 1 failure, 7 ignored = (2 assumptions) Total time: 24 minutes 32 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording test results Description set: Java: 32bit/jdk1.6.0_37 -server -XX:+UseParallelGC Email was triggered for: Failure Sending email for trigger: Failure ------=_Part_103_103521297.1352398452134 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org ------=_Part_103_103521297.1352398452134--