Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 67CDDD26A for ; Tue, 5 Mar 2013 07:06:41 +0000 (UTC) Received: (qmail 39805 invoked by uid 500); 5 Mar 2013 07:06:39 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 39690 invoked by uid 500); 5 Mar 2013 07:06:38 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 39673 invoked by uid 99); 5 Mar 2013 07:06:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Mar 2013 07:06:38 +0000 X-ASF-Spam-Status: No, hits=0.0 required=5.0 tests=NORMAL_HTTP_TO_IP,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jenkins@thetaphi.de designates 188.138.97.18 as permitted sender) Received: from [188.138.97.18] (HELO mail.sd-datasolutions.de) (188.138.97.18) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Mar 2013 07:06:15 +0000 Received: from serv1 (localhost.localdomain [127.0.0.1]) by mail.sd-datasolutions.de (Postfix) with ESMTP id CD23014AA054 for ; Tue, 5 Mar 2013 07:05:54 +0000 (UTC) Date: Tue, 5 Mar 2013 07:05:03 +0000 (UTC) From: Policeman Jenkins Server To: dev@lucene.apache.org Message-ID: <292635130.17.1362467154811.JavaMail.jenkins@serv1.sd-datasolutions.de> In-Reply-To: <918990260.15.1362458945642.JavaMail.jenkins@serv1.sd-datasolutions.de> References: <918990260.15.1362458945642.JavaMail.jenkins@serv1.sd-datasolutions.de> Subject: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jrockit-jdk1.6.0_33-R28.2.4-4.1.0) - Build # 4576 - Still Failing! MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_16_1303718807.1362467103398" X-Jenkins-Job: Lucene-Solr-trunk-Linux X-Jenkins-Result: FAILURE X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_16_1303718807.1362467103398 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/4576/ Java: 64bit/jrockit-jdk1.6.0_33-R28.2.4-4.1.0 -XnoOpt 1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLea= derTest Error Message: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLe= aderTest: 1) Thread[id=3D2123, name=3DTEST-ChaosMonkeySafeLeaderTest.te= stDistribSearch-seed#[E22BFD0C90735963]-EventThread, state=3DTIMED_WAITING,= group=3DTGRP-ChaosMonkeySafeLeaderTest] at java.lang.Thread.sleep(= Native Method) at org.apache.solr.common.cloud.ZkStateReader.getLea= derRetry(ZkStateReader.java:428) at org.apache.solr.common.cloud.Zk= StateReader.getLeaderUrl(ZkStateReader.java:404) at org.apache.solr= .cloud.ZkController.getLeader(ZkController.java:833) at org.apache.= solr.cloud.ZkController.register(ZkController.java:766) at org.apac= he.solr.cloud.ZkController$1.command(ZkController.java:212) at org.= apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:= 117) at org.apache.solr.common.cloud.DefaultConnectionStrategy.reco= nnect(DefaultConnectionStrategy.java:46) at org.apache.solr.common.= cloud.ConnectionManager.process(ConnectionManager.java:91) at org.a= pache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) = at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:49= 5) Stack Trace: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SU= ITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:=20 1) Thread[id=3D2123, name=3DTEST-ChaosMonkeySafeLeaderTest.testDistribSe= arch-seed#[E22BFD0C90735963]-EventThread, state=3DTIMED_WAITING, group=3DTG= RP-ChaosMonkeySafeLeaderTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStat= eReader.java:428) at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateR= eader.java:404) at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:8= 33) at org.apache.solr.cloud.ZkController.register(ZkController.java:76= 6) at org.apache.solr.cloud.ZkController$1.command(ZkController.java:2= 12) at org.apache.solr.common.cloud.ConnectionManager$1.update(Connecti= onManager.java:117) at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect= (DefaultConnectionStrategy.java:46) at org.apache.solr.common.cloud.ConnectionManager.process(Connectio= nManager.java:91) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientC= nxn.java:519) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:= 495) =09at __randomizedtesting.SeedInfo.seed([E22BFD0C90735963]:0) Build Log: [...truncated 8512 lines...] [junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest [junit4:junit4] 2> 0 T1904 oas.BaseDistributedSearchTestCase.initHostCont= ext Setting hostContext system property: /um_xr/ [junit4:junit4] 2> 20 T1904 oas.SolrTestCaseJ4.setUp ###Starting testDist= ribSearch [junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-So= lr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeade= rTest-1362466196085 [junit4:junit4] 2> 21 T1904 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4:junit4] 2> 22 T1905 oasc.ZkTestServer$ZKServerMain.runFromConfig = Starting server [junit4:junit4] 2> 22 T1905 oazs.ZooKeeperServer.setTickTime tickTime set= to 1000 [junit4:junit4] 2> 22 T1905 oazs.ZooKeeperServer.setMinSessionTimeout min= SessionTimeout set to -1 [junit4:junit4] 2> 23 T1905 oazs.ZooKeeperServer.setMaxSessionTimeout max= SessionTimeout set to -1 [junit4:junit4] 2> 23 T1905 oazs.NIOServerCnxnFactory.configure binding t= o port 0.0.0.0/0.0.0.0:0 [junit4:junit4] 2> 23 T1905 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 t= o /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/t= est/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466196085/zook= eeper/server1/data/version-2/snapshot.0 [junit4:junit4] 2> 122 T1904 oasc.ZkTestServer.run start zk server on por= t:33229 [junit4:junit4] 2> 122 T1904 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:33229 sessionTimeout=3D10000 watcher=3Dorg= .apache.solr.common.cloud.ConnectionManager@1cf5835a [junit4:junit4] 2> 123 T1910 oaz.ClientCnxn$SendThread.logStartConnect Op= ening socket connection to server localhost.localdomain/127.0.0.1:33229. Wi= ll not attempt to authenticate using SASL (access denied (javax.security.au= th.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 123 T1904 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 124 T1910 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost.localdomain/127.0.0.1:33229, initi= ating session [junit4:junit4] 2> 124 T1906 oazs.NIOServerCnxnFactory.run Accepted socke= t connection from /127.0.0.1:41302 [junit4:junit4] 2> 124 T1906 oazs.ZooKeeperServer.processConnectRequest C= lient attempting to establish new session at /127.0.0.1:41302 [junit4:junit4] 2> 124 T1908 oazsp.FileTxnLog.append Creating new log fil= e: log.1 [junit4:junit4] 2> 186 T1908 oazs.ZooKeeperServer.finishSessionInit Estab= lished session 0x13d39504a780000 with negotiated timeout 10000 for client /= 127.0.0.1:41302 [junit4:junit4] 2> 186 T1910 oaz.ClientCnxn$SendThread.onConnected Sessio= n establishment complete on server localhost.localdomain/127.0.0.1:33229, s= essionid =3D 0x13d39504a780000, negotiated timeout =3D 10000 [junit4:junit4] 2> 186 T1911 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@1cf5835a name:ZooKeeperConnectio= n Watcher:127.0.0.1:33229 got event WatchedEvent state:SyncConnected type:N= one path:null path:null type:None [junit4:junit4] 2> 187 T1904 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 187 T1904 oascc.SolrZkClient.makePath makePath: /solr [junit4:junit4] 2> 190 T1909 oazs.PrepRequestProcessor.pRequest2Txn Proce= ssed session termination for sessionid: 0x13d39504a780000 [junit4:junit4] 2> 191 T1911 oaz.ClientCnxn$EventThread.run EventThread s= hut down [junit4:junit4] 2> 191 T1904 oaz.ZooKeeper.close Session: 0x13d39504a7800= 00 closed [junit4:junit4] 2> 191 T1906 oazs.NIOServerCnxn.closeSock Closed socket c= onnection for client /127.0.0.1:41302 which had sessionid 0x13d39504a780000 [junit4:junit4] 2> 191 T1904 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:33229/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@1cf5e601 [junit4:junit4] 2> 192 T1912 oaz.ClientCnxn$SendThread.logStartConnect Op= ening socket connection to server localhost.localdomain/127.0.0.1:33229. Wi= ll not attempt to authenticate using SASL (access denied (javax.security.au= th.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 192 T1904 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 193 T1912 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost.localdomain/127.0.0.1:33229, initi= ating session [junit4:junit4] 2> 193 T1906 oazs.NIOServerCnxnFactory.run Accepted socke= t connection from /127.0.0.1:41303 [junit4:junit4] 2> 193 T1906 oazs.ZooKeeperServer.processConnectRequest C= lient attempting to establish new session at /127.0.0.1:41303 [junit4:junit4] 2> 194 T1908 oazs.ZooKeeperServer.finishSessionInit Estab= lished session 0x13d39504a780001 with negotiated timeout 10000 for client /= 127.0.0.1:41303 [junit4:junit4] 2> 194 T1912 oaz.ClientCnxn$SendThread.onConnected Sessio= n establishment complete on server localhost.localdomain/127.0.0.1:33229, s= essionid =3D 0x13d39504a780001, negotiated timeout =3D 10000 [junit4:junit4] 2> 194 T1913 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@1cf5e601 name:ZooKeeperConnectio= n Watcher:127.0.0.1:33229/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 194 T1904 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 194 T1904 oascc.SolrZkClient.makePath makePath: /colle= ctions/collection1 [junit4:junit4] 2> 197 T1904 oascc.SolrZkClient.makePath makePath: /colle= ctions/collection1/shards [junit4:junit4] 2> 198 T1904 oascc.SolrZkClient.makePath makePath: /colle= ctions/control_collection [junit4:junit4] 2> 200 T1904 oascc.SolrZkClient.makePath makePath: /colle= ctions/control_collection/shards [junit4:junit4] 2> 202 T1904 oasc.AbstractZkTestCase.putConfig put /mnt/s= sd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-file= s/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xm= l [junit4:junit4] 2> 202 T1904 oascc.SolrZkClient.makePath makePath: /confi= gs/conf1/solrconfig.xml [junit4:junit4] 2> 209 T1904 oasc.AbstractZkTestCase.putConfig put /mnt/s= sd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-file= s/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml [junit4:junit4] 2> 210 T1904 oascc.SolrZkClient.makePath makePath: /confi= gs/conf1/schema.xml [junit4:junit4] 2> 213 T1904 oasc.AbstractZkTestCase.putConfig put /mnt/s= sd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-file= s/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt [junit4:junit4] 2> 213 T1904 oascc.SolrZkClient.makePath makePath: /confi= gs/conf1/stopwords.txt [junit4:junit4] 2> 215 T1904 oasc.AbstractZkTestCase.putConfig put /mnt/s= sd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-file= s/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt [junit4:junit4] 2> 215 T1904 oascc.SolrZkClient.makePath makePath: /confi= gs/conf1/protwords.txt [junit4:junit4] 2> 217 T1904 oasc.AbstractZkTestCase.putConfig put /mnt/s= sd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-file= s/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml [junit4:junit4] 2> 217 T1904 oascc.SolrZkClient.makePath makePath: /confi= gs/conf1/currency.xml [junit4:junit4] 2> 219 T1904 oasc.AbstractZkTestCase.putConfig put /mnt/s= sd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-file= s/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exc= hange-rates.json [junit4:junit4] 2> 220 T1904 oascc.SolrZkClient.makePath makePath: /confi= gs/conf1/open-exchange-rates.json [junit4:junit4] 2> 222 T1904 oasc.AbstractZkTestCase.putConfig put /mnt/s= sd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-file= s/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mappi= ng-ISOLatin1Accent.txt [junit4:junit4] 2> 222 T1904 oascc.SolrZkClient.makePath makePath: /confi= gs/conf1/mapping-ISOLatin1Accent.txt [junit4:junit4] 2> 224 T1904 oasc.AbstractZkTestCase.putConfig put /mnt/s= sd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-file= s/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt [junit4:junit4] 2> 224 T1904 oascc.SolrZkClient.makePath makePath: /confi= gs/conf1/old_synonyms.txt [junit4:junit4] 2> 226 T1904 oasc.AbstractZkTestCase.putConfig put /mnt/s= sd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-file= s/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt [junit4:junit4] 2> 227 T1904 oascc.SolrZkClient.makePath makePath: /confi= gs/conf1/synonyms.txt [junit4:junit4] 2> 228 T1909 oazs.PrepRequestProcessor.pRequest2Txn Proce= ssed session termination for sessionid: 0x13d39504a780001 [junit4:junit4] 2> 229 T1913 oaz.ClientCnxn$EventThread.run EventThread s= hut down [junit4:junit4] 2> 229 T1904 oaz.ZooKeeper.close Session: 0x13d39504a7800= 01 closed [junit4:junit4] 2> 229 T1906 oazs.NIOServerCnxn.closeSock Closed socket c= onnection for client /127.0.0.1:41303 which had sessionid 0x13d39504a780001 [junit4:junit4] 2> 288 T1904 oejs.Server.doStart jetty-8.1.8.v20121106 [junit4:junit4] 2> 295 T1904 oejs.AbstractConnector.doStart Started Selec= tChannelConnector@127.0.0.1:53417 [junit4:junit4] 2> 295 T1904 oass.SolrDispatchFilter.init SolrDispatchFil= ter.init() [junit4:junit4] 2> 296 T1904 oasc.SolrResourceLoader.locateSolrHome JNDI = not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 296 T1904 oasc.SolrResourceLoader.locateSolrHome using= system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-tru= nk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySa= feLeaderTest-controljetty-1362466196293 [junit4:junit4] 2> 296 T1904 oasc.CoreContainer$Initializer.initialize lo= oking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-controljetty-1362466196293/solr.xml [junit4:junit4] 2> 296 T1904 oasc.CoreContainer. New CoreContainer = 488124082 [junit4:junit4] 2> 297 T1904 oasc.CoreContainer.load Loading CoreContaine= r using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr= /build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-= controljetty-1362466196293/' [junit4:junit4] 2> 297 T1904 oasc.SolrResourceLoader. new SolrResou= rceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linu= x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeade= rTest-controljetty-1362466196293/' [junit4:junit4] 2> 313 T1904 oashc.HttpShardHandlerFactory.getParameter S= etting socketTimeout to: 120000 [junit4:junit4] 2> 314 T1904 oashc.HttpShardHandlerFactory.getParameter S= etting urlScheme to: http:// [junit4:junit4] 2> 314 T1904 oashc.HttpShardHandlerFactory.getParameter S= etting connTimeout to: 15000 [junit4:junit4] 2> 314 T1904 oashc.HttpShardHandlerFactory.getParameter S= etting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 314 T1904 oashc.HttpShardHandlerFactory.getParameter S= etting corePoolSize to: 0 [junit4:junit4] 2> 315 T1904 oashc.HttpShardHandlerFactory.getParameter S= etting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 315 T1904 oashc.HttpShardHandlerFactory.getParameter S= etting maxThreadIdleTime to: 5 [junit4:junit4] 2> 315 T1904 oashc.HttpShardHandlerFactory.getParameter S= etting sizeOfQueue to: -1 [junit4:junit4] 2> 316 T1904 oashc.HttpShardHandlerFactory.getParameter S= etting fairnessPolicy to: false [junit4:junit4] 2> 316 T1904 oascsi.HttpClientUtil.createClient Creating = new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000&s= ocketTimeout=3D120000&connTimeout=3D15000&retry=3Dfalse [junit4:junit4] 2> 323 T1904 oasc.CoreContainer.load Registering Log List= ener [junit4:junit4] 2> 337 T1904 oasc.CoreContainer.initZooKeeper Zookeeper c= lient=3D127.0.0.1:33229/solr [junit4:junit4] 2> 337 T1904 oasc.ZkController.checkChrootPath zkHost inc= ludes chroot [junit4:junit4] 2> 338 T1904 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:33229 sessionTimeout=3D60000 watcher=3Dorg= .apache.solr.common.cloud.ConnectionManager@1d0b9b8d [junit4:junit4] 2> 339 T1923 oaz.ClientCnxn$SendThread.logStartConnect Op= ening socket connection to server localhost.localdomain/127.0.0.1:33229. Wi= ll not attempt to authenticate using SASL (access denied (javax.security.au= th.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 338 T1904 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 339 T1923 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost.localdomain/127.0.0.1:33229, initi= ating session [junit4:junit4] 2> 339 T1906 oazs.NIOServerCnxnFactory.run Accepted socke= t connection from /127.0.0.1:41304 [junit4:junit4] 2> 339 T1906 oazs.ZooKeeperServer.processConnectRequest C= lient attempting to establish new session at /127.0.0.1:41304 [junit4:junit4] 2> 340 T1908 oazs.ZooKeeperServer.finishSessionInit Estab= lished session 0x13d39504a780002 with negotiated timeout 20000 for client /= 127.0.0.1:41304 [junit4:junit4] 2> 340 T1923 oaz.ClientCnxn$SendThread.onConnected Sessio= n establishment complete on server localhost.localdomain/127.0.0.1:33229, s= essionid =3D 0x13d39504a780002, negotiated timeout =3D 20000 [junit4:junit4] 2> 340 T1924 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@1d0b9b8d name:ZooKeeperConnectio= n Watcher:127.0.0.1:33229 got event WatchedEvent state:SyncConnected type:N= one path:null path:null type:None [junit4:junit4] 2> 340 T1904 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 341 T1909 oazs.PrepRequestProcessor.pRequest2Txn Proce= ssed session termination for sessionid: 0x13d39504a780002 [junit4:junit4] 2> 342 T1924 oaz.ClientCnxn$EventThread.run EventThread s= hut down [junit4:junit4] 2> 342 T1904 oaz.ZooKeeper.close Session: 0x13d39504a7800= 02 closed [junit4:junit4] 2> 342 T1906 oazs.NIOServerCnxn.closeSock Closed socket c= onnection for client /127.0.0.1:41304 which had sessionid 0x13d39504a780002 [junit4:junit4] 2> 342 T1904 oascsi.HttpClientUtil.createClient Creating = new http client, config:maxConnections=3D500&maxConnectionsPerHost=3D16&soc= ketTimeout=3D120000&connTimeout=3D15000 [junit4:junit4] 2> 345 T1904 oaz.ZooKeeper. Initiating client conne= ction, connectString=3D127.0.0.1:33229/solr sessionTimeout=3D30000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@1d06b906 [junit4:junit4] 2> 346 T1925 oaz.ClientCnxn$SendThread.logStartConnect Op= ening socket connection to server localhost.localdomain/127.0.0.1:33229. Wi= ll not attempt to authenticate using SASL (access denied (javax.security.au= th.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 346 T1904 oascc.ConnectionManager.waitForConnected Wai= ting for client to connect to ZooKeeper [junit4:junit4] 2> 346 T1925 oaz.ClientCnxn$SendThread.primeConnection So= cket connection established to localhost.localdomain/127.0.0.1:33229, initi= ating session [junit4:junit4] 2> 346 T1906 oazs.NIOServerCnxnFactory.run Accepted socke= t connection from /127.0.0.1:41305 [junit4:junit4] 2> 346 T1906 oazs.ZooKeeperServer.processConnectRequest C= lient attempting to establish new session at /127.0.0.1:41305 [junit4:junit4] 2> 347 T1908 oazs.ZooKeeperServer.finishSessionInit Estab= lished session 0x13d39504a780003 with negotiated timeout 20000 for client /= 127.0.0.1:41305 [junit4:junit4] 2> 347 T1925 oaz.ClientCnxn$SendThread.onConnected Sessio= n establishment complete on server localhost.localdomain/127.0.0.1:33229, s= essionid =3D 0x13d39504a780003, negotiated timeout =3D 20000 [junit4:junit4] 2> 347 T1926 oascc.ConnectionManager.process Watcher org.= apache.solr.common.cloud.ConnectionManager@1d06b906 name:ZooKeeperConnectio= n Watcher:127.0.0.1:33229/solr got event WatchedEvent state:SyncConnected t= ype:None path:null path:null type:None [junit4:junit4] 2> 348 T1904 oascc.ConnectionManager.waitForConnected Cli= ent is connected to ZooKeeper [junit4:junit4] 2> 349 T1909 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13d39504a780003 type:crea= te cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Erro= r:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 350 T1904 oascc.SolrZkClient.makePath makePath: /live_= nodes [junit4:junit4] 2> 351 T1904 oasc.ZkController.createEphemeralLiveNode Re= gister node as live in ZooKeeper:/live_nodes/127.0.0.1:53417_um_xr [junit4:junit4] 2> 351 T1909 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13d39504a780003 type:dele= te cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/12= 7.0.0.1:53417_um_xr Error:KeeperErrorCode =3D NoNode for /solr/live_nodes/1= 27.0.0.1:53417_um_xr [junit4:junit4] 2> 352 T1904 oascc.SolrZkClient.makePath makePath: /live_= nodes/127.0.0.1:53417_um_xr [junit4:junit4] 2> 354 T1904 oascc.SolrZkClient.makePath makePath: /overs= eer_elect/election [junit4:junit4] 2> 358 T1909 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13d39504a780003 type:dele= te cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_ele= ct/leader Error:KeeperErrorCode =3D NoNode for /solr/overseer_elect/leader [junit4:junit4] 2> 359 T1904 oascc.SolrZkClient.makePath makePath: /overs= eer_elect/leader [junit4:junit4] 2> 361 T1904 oasc.Overseer.start Overseer (id=3D892905846= 26823171-127.0.0.1:53417_um_xr-n_0000000000) starting [junit4:junit4] 2> 361 T1909 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13d39504a780003 type:crea= te cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Err= or:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 362 T1909 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13d39504a780003 type:crea= te cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Err= or:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 362 T1909 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13d39504a780003 type:crea= te cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Err= or:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 363 T1909 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13d39504a780003 type:crea= te cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Err= or:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 364 T1928 oasc.OverseerCollectionProcessor.run Process= current queue of collection creations [junit4:junit4] 2> 364 T1904 oascc.SolrZkClient.makePath makePath: /clust= erstate.json [junit4:junit4] 2> 366 T1904 oascc.ZkStateReader.createClusterStateWatche= rsAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 367 T1927 oasc.Overseer$ClusterStateUpdater.run Starti= ng to work on the main queue [junit4:junit4] 2> 370 T1929 oasc.CoreContainer.create Creating SolrCore = 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-tru= nk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySa= feLeaderTest-controljetty-1362466196293/collection1 [junit4:junit4] 2> 370 T1929 oasc.ZkController.createCollectionZkNode Che= ck for collection zkNode:control_collection [junit4:junit4] 2> 371 T1929 oasc.ZkController.createCollectionZkNode Col= lection zkNode exists [junit4:junit4] 2> 371 T1929 oasc.ZkController.readConfigName Load collec= tion config from:/collections/control_collection [junit4:junit4] 2> 372 T1929 oasc.SolrResourceLoader. new SolrResou= rceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linu= x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeade= rTest-controljetty-1362466196293/collection1/' [junit4:junit4] 2> 372 T1929 oasc.SolrResourceLoader.replaceClassLoader A= dding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/s= olr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljet= ty-1362466196293/collection1/lib/README' to classloader [junit4:junit4] 2> 372 T1929 oasc.SolrResourceLoader.replaceClassLoader A= dding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/s= olr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljet= ty-1362466196293/collection1/lib/classes/' to classloader [junit4:junit4] 2> 414 T1929 oasc.SolrConfig. Using Lucene MatchVer= sion: LUCENE_50 [junit4:junit4] 2> 457 T1929 oasc.SolrConfig. Loaded SolrConfig: so= lrconfig.xml [junit4:junit4] 2> 458 T1929 oass.IndexSchema.readSchema Reading Solr Sch= ema [junit4:junit4] 2> 463 T1929 oass.IndexSchema.readSchema Schema name=3Dte= st [junit4:junit4] 2> 927 T1929 oass.OpenExchangeRatesOrgProvider.init Initi= alized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 934 T1929 oass.IndexSchema.readSchema default search f= ield in schema is text [junit4:junit4] 2> 937 T1929 oass.IndexSchema.readSchema unique key field= : id [junit4:junit4] 2> 946 T1929 oass.FileExchangeRateProvider.reload Reloadi= ng exchange rates from file currency.xml [junit4:junit4] 2> 949 T1929 oass.FileExchangeRateProvider.reload Reloadi= ng exchange rates from file currency.xml [junit4:junit4] 2> 952 T1929 oass.OpenExchangeRatesOrgProvider.reload Rel= oading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 953 T1929 oass.OpenExchangeRatesOrgProvider$OpenExchan= geRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 953 T1929 oass.OpenExchangeRatesOrgProvider$OpenExchan= geRates. WARNING Expected key, got STRING [junit4:junit4] 2> 953 T1929 oass.OpenExchangeRatesOrgProvider.reload Rel= oading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 954 T1929 oass.OpenExchangeRatesOrgProvider$OpenExchan= geRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 954 T1929 oass.OpenExchangeRatesOrgProvider$OpenExchan= geRates. WARNING Expected key, got STRING [junit4:junit4] 2> 955 T1929 oasc.SolrCore. [collection1] Opening n= ew SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/buil= d/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-contr= oljetty-1362466196293/collection1/, dataDir=3D./org.apache.solr.cloud.Chaos= MonkeySafeLeaderTest-1362466196085/control/data/ [junit4:junit4] 2> 955 T1929 oasc.JmxMonitoredMap. JMX monitoring i= s enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBe= anServer@1c1fa491 [junit4:junit4] 2> 955 T1929 oasc.SolrCore.initDirectoryFactory solr.Stan= dardDirectoryFactory [junit4:junit4] 2> 961 T1929 oasc.CachingDirectoryFactory.get return new = directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-136246= 6196085/control/data forceNew:false [junit4:junit4] 2> 962 T1929 oasc.CachingDirectoryFactory.close Releasing= directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/so= lr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-136246619= 6085/control/data [junit4:junit4] 2> 962 T1929 oasc.SolrCore.getNewIndexDir New index direc= tory detected: old=3Dnull new=3D./org.apache.solr.cloud.ChaosMonkeySafeLead= erTest-1362466196085/control/data/index/ [junit4:junit4] 2> 962 T1929 oasc.SolrCore.initIndex WARNING [collection1= ] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1= 362466196085/control/data/index' doesn't exist. Creating new index... [junit4:junit4] 2> 962 T1929 oasc.CachingDirectoryFactory.get return new = directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-136246= 6196085/control/data/index forceNew:false [junit4:junit4] 2> 965 T1929 oasc.SolrDeletionPolicy.onCommit SolrDeletio= nPolicy.onCommit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/control/data/index,segFN=3Dsegments_1,gener= ation=3D1,filenames=3D[segments_1] [junit4:junit4] 2> 965 T1929 oasc.SolrDeletionPolicy.updateCommits newest= commit =3D 1[segments_1] [junit4:junit4] 2> 966 T1929 oasc.CachingDirectoryFactory.close Releasing= directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/so= lr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-136246619= 6085/control/data/index [junit4:junit4] 2> 967 T1929 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "de= dupe" [junit4:junit4] 2> 967 T1929 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "st= ored_sig" [junit4:junit4] 2> 967 T1929 oasup.UpdateRequestProcessorChain.init inser= ting DistributedUpdateProcessorFactory into updateRequestProcessorChain "di= strib-dup-test-chain-implicit" [junit4:junit4] 2> 967 T1929 oasc.RequestHandlers.initHandlersFromConfig = created standard: solr.StandardRequestHandler [junit4:junit4] 2> 968 T1929 oasc.RequestHandlers.initHandlersFromConfig = created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 968 T1929 oasc.RequestHandlers.initHandlersFromConfig = adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 968 T1929 oasc.RequestHandlers.initHandlersFromConfig = created /replication: solr.ReplicationHandler [junit4:junit4] 2> 969 T1929 oasc.RequestHandlers.initHandlersFromConfig = created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 969 T1929 oasc.RequestHandlers.initHandlersFromConfig = created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 971 T1929 oashl.XMLLoader.init xsltCacheLifetimeSecond= s=3D60 [junit4:junit4] 2> 973 T1929 oasc.CachingDirectoryFactory.close Releasing= directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/so= lr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-136246619= 6085/control/data [junit4:junit4] 2> 974 T1929 oass.SolrIndexSearcher. Opening Search= er@1c5f3ebe main [junit4:junit4] 2> 974 T1929 oasu.CommitTracker. Hard AutoCommit: d= isabled [junit4:junit4] 2> 974 T1929 oasu.CommitTracker. Soft AutoCommit: d= isabled [junit4:junit4] 2> 978 T1929 oasc.CachingDirectoryFactory.close Releasing= directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/so= lr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-136246619= 6085/control/data [junit4:junit4] 2> 979 T1930 oasc.SolrCore.registerSearcher [collection1]= Registered new searcher Searcher@1c5f3ebe main{StandardDirectoryReader(seg= ments_1:1)} [junit4:junit4] 2> 979 T1929 oasc.ZkController.publish publishing core=3D= collection1 state=3Ddown [junit4:junit4] 2> 979 T1929 oasc.ZkController.publish numShards not foun= d on descriptor - reading it from system property [junit4:junit4] 2> 981 T1909 oazs.PrepRequestProcessor.pRequest Got user-= level KeeperException when processing sessionid:0x13d39504a780003 type:crea= te cxid:0x50 zxid:0x29 txntype:-1 reqpath:n/a Error Path:/solr/overseer/que= ue Error:KeeperErrorCode =3D NoNode for /solr/overseer/queue [junit4:junit4] 2> 1870 T1927 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 1871 T1927 oasc.Overseer$ClusterStateUpdater.updateSta= te Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "shard":null, [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"down", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"control_collection", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:53417_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:53417/um_xr"} [junit4:junit4] 2> 1871 T1927 oasc.Overseer$ClusterStateUpdater.createCol= lection Create collection control_collection with numShards 1 [junit4:junit4] 2> 1875 T1927 oasc.Overseer$ClusterStateUpdater.updateSta= te Assigning new node to shard shard=3Dshard1 [junit4:junit4] 2> 1875 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780003 type:cre= ate cxid:0x58 zxid:0x2c txntype:-1 reqpath:n/a Error Path:/solr/overseer/qu= eue-work Error:KeeperErrorCode =3D NoNode for /solr/overseer/queue-work [junit4:junit4] 2> 1879 T1926 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 1) [junit4:junit4] 2> 1984 T1929 oasc.CoreContainer.registerCore registering= core: collection1 [junit4:junit4] 2> 1984 T1929 oasc.ZkController.register Register replica= - core:collection1 address:http://127.0.0.1:53417/um_xr collection:control= _collection shard:shard1 [junit4:junit4] 2> 1985 T1929 oascc.SolrZkClient.makePath makePath: /coll= ections/control_collection/leader_elect/shard1/election [junit4:junit4] 2> 1989 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780003 type:del= ete cxid:0x72 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/solr/collections= /control_collection/leaders Error:KeeperErrorCode =3D NoNode for /solr/coll= ections/control_collection/leaders [junit4:junit4] 2> 1990 T1929 oasc.ShardLeaderElectionContext.runLeaderPr= ocess Running the leader process. [junit4:junit4] 2> 1990 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780003 type:cre= ate cxid:0x73 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 1991 T1929 oasc.ShardLeaderElectionContext.waitForRepl= icasToComeUp Enough replicas found to continue. [junit4:junit4] 2> 1991 T1929 oasc.ShardLeaderElectionContext.runLeaderPr= ocess I may be the new leader - try and sync [junit4:junit4] 2> 1992 T1929 oasc.SyncStrategy.sync Sync replicas to htt= p://127.0.0.1:53417/um_xr/collection1/ [junit4:junit4] 2> 1992 T1929 oasc.SyncStrategy.syncReplicas Sync Success= - now sync replicas to me [junit4:junit4] 2> 1992 T1929 oasc.SyncStrategy.syncToMe http://127.0.0.1= :53417/um_xr/collection1/ has no replicas [junit4:junit4] 2> 1992 T1929 oasc.ShardLeaderElectionContext.runLeaderPr= ocess I am the new leader: http://127.0.0.1:53417/um_xr/collection1/ [junit4:junit4] 2> 1992 T1929 oascc.SolrZkClient.makePath makePath: /coll= ections/control_collection/leaders/shard1 [junit4:junit4] 2> 1995 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780003 type:cre= ate cxid:0x7d zxid:0x3b txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 3383 T1927 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 3391 T1926 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 1) [junit4:junit4] 2> 3403 T1929 oasc.ZkController.register We are http://12= 7.0.0.1:53417/um_xr/collection1/ and leader is http://127.0.0.1:53417/um_xr= /collection1/ [junit4:junit4] 2> 3404 T1929 oasc.ZkController.register No LogReplay nee= ded for core=3Dcollection1 baseURL=3Dhttp://127.0.0.1:53417/um_xr [junit4:junit4] 2> 3404 T1929 oasc.ZkController.checkRecovery I am the le= ader, no recovery necessary [junit4:junit4] 2> 3404 T1929 oasc.ZkController.publish publishing core= =3Dcollection1 state=3Dactive [junit4:junit4] 2> 3405 T1929 oasc.ZkController.publish numShards not fou= nd on descriptor - reading it from system property [junit4:junit4] 2> 3408 T1929 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 3410 T1904 oass.SolrDispatchFilter.init user.dir=3D/mn= t/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J= 1 [junit4:junit4] 2> 3410 T1904 oass.SolrDispatchFilter.init SolrDispatchFi= lter.init() done [junit4:junit4] 2> 3411 T1904 oascsi.HttpClientUtil.createClient Creating= new http client, config:maxConnections=3D128&maxConnectionsPerHost=3D32&fo= llowRedirects=3Dfalse [junit4:junit4] 2> 3416 T1904 oascsi.HttpClientUtil.createClient Creating= new http client, config: [junit4:junit4] 2> 3417 T1904 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:33229/solr sessionTimeout=3D10000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@1c720fdb [junit4:junit4] 2> 3418 T1932 oaz.ClientCnxn$SendThread.logStartConnect O= pening socket connection to server localhost.localdomain/127.0.0.1:33229. W= ill not attempt to authenticate using SASL (access denied (javax.security.a= uth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 3418 T1904 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 3418 T1932 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:33229, init= iating session [junit4:junit4] 2> 3418 T1906 oazs.NIOServerCnxnFactory.run Accepted sock= et connection from /127.0.0.1:41306 [junit4:junit4] 2> 3419 T1906 oazs.ZooKeeperServer.processConnectRequest = Client attempting to establish new session at /127.0.0.1:41306 [junit4:junit4] 2> 3420 T1908 oazs.ZooKeeperServer.finishSessionInit Esta= blished session 0x13d39504a780004 with negotiated timeout 10000 for client = /127.0.0.1:41306 [junit4:junit4] 2> 3420 T1932 oaz.ClientCnxn$SendThread.onConnected Sessi= on establishment complete on server localhost.localdomain/127.0.0.1:33229, = sessionid =3D 0x13d39504a780004, negotiated timeout =3D 10000 [junit4:junit4] 2> 3420 T1933 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@1c720fdb name:ZooKeeperConnecti= on Watcher:127.0.0.1:33229/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 3421 T1904 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 3422 T1904 oascc.ZkStateReader.createClusterStateWatch= ersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 3424 T1904 oasc.ChaosMonkey.monkeyLog monkey: init - e= xpire sessions:true cause connection loss:true [junit4:junit4] 2> 3485 T1904 oejs.Server.doStart jetty-8.1.8.v20121106 [junit4:junit4] 2> 3487 T1904 oejs.AbstractConnector.doStart Started Sele= ctChannelConnector@127.0.0.1:55383 [junit4:junit4] 2> 3487 T1904 oass.SolrDispatchFilter.init SolrDispatchFi= lter.init() [junit4:junit4] 2> 3488 T1904 oasc.SolrResourceLoader.locateSolrHome JNDI= not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 3488 T1904 oasc.SolrResourceLoader.locateSolrHome usin= g system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-tr= unk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyS= afeLeaderTest-jetty1-1362466199488 [junit4:junit4] 2> 3488 T1904 oasc.CoreContainer$Initializer.initialize l= ooking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-L= inux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLe= aderTest-jetty1-1362466199488/solr.xml [junit4:junit4] 2> 3489 T1904 oasc.CoreContainer. New CoreContainer= 478254280 [junit4:junit4] 2> 3489 T1904 oasc.CoreContainer.load Loading CoreContain= er using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/sol= r/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest= -jetty1-1362466199488/' [junit4:junit4] 2> 3489 T1904 oasc.SolrResourceLoader. new SolrReso= urceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Lin= ux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLead= erTest-jetty1-1362466199488/' [junit4:junit4] 2> 3506 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting socketTimeout to: 120000 [junit4:junit4] 2> 3506 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting urlScheme to: http:// [junit4:junit4] 2> 3506 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting connTimeout to: 15000 [junit4:junit4] 2> 3507 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 3507 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting corePoolSize to: 0 [junit4:junit4] 2> 3507 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 3508 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting maxThreadIdleTime to: 5 [junit4:junit4] 2> 3508 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting sizeOfQueue to: -1 [junit4:junit4] 2> 3508 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting fairnessPolicy to: false [junit4:junit4] 2> 3508 T1904 oascsi.HttpClientUtil.createClient Creating= new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000&= socketTimeout=3D120000&connTimeout=3D15000&retry=3Dfalse [junit4:junit4] 2> 3517 T1904 oasc.CoreContainer.load Registering Log Lis= tener [junit4:junit4] 2> 3530 T1904 oasc.CoreContainer.initZooKeeper Zookeeper = client=3D127.0.0.1:33229/solr [junit4:junit4] 2> 3530 T1904 oasc.ZkController.checkChrootPath zkHost in= cludes chroot [junit4:junit4] 2> 3531 T1904 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:33229 sessionTimeout=3D60000 watcher=3Dor= g.apache.solr.common.cloud.ConnectionManager@1c9a2c41 [junit4:junit4] 2> 3532 T1943 oaz.ClientCnxn$SendThread.logStartConnect O= pening socket connection to server localhost.localdomain/127.0.0.1:33229. W= ill not attempt to authenticate using SASL (access denied (javax.security.a= uth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 3532 T1904 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 3532 T1943 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:33229, init= iating session [junit4:junit4] 2> 3532 T1906 oazs.NIOServerCnxnFactory.run Accepted sock= et connection from /127.0.0.1:41307 [junit4:junit4] 2> 3532 T1906 oazs.ZooKeeperServer.processConnectRequest = Client attempting to establish new session at /127.0.0.1:41307 [junit4:junit4] 2> 3533 T1908 oazs.ZooKeeperServer.finishSessionInit Esta= blished session 0x13d39504a780005 with negotiated timeout 20000 for client = /127.0.0.1:41307 [junit4:junit4] 2> 3533 T1943 oaz.ClientCnxn$SendThread.onConnected Sessi= on establishment complete on server localhost.localdomain/127.0.0.1:33229, = sessionid =3D 0x13d39504a780005, negotiated timeout =3D 20000 [junit4:junit4] 2> 3533 T1944 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@1c9a2c41 name:ZooKeeperConnecti= on Watcher:127.0.0.1:33229 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 3534 T1904 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 3534 T1909 oazs.PrepRequestProcessor.pRequest2Txn Proc= essed session termination for sessionid: 0x13d39504a780005 [junit4:junit4] 2> 3535 T1944 oaz.ClientCnxn$EventThread.run EventThread = shut down [junit4:junit4] 2> 3535 T1904 oaz.ZooKeeper.close Session: 0x13d39504a780= 005 closed [junit4:junit4] 2> 3535 T1906 oazs.NIOServerCnxn.closeSock Closed socket = connection for client /127.0.0.1:41307 which had sessionid 0x13d39504a78000= 5 [junit4:junit4] 2> 3535 T1904 oascsi.HttpClientUtil.createClient Creating= new http client, config:maxConnections=3D500&maxConnectionsPerHost=3D16&so= cketTimeout=3D120000&connTimeout=3D15000 [junit4:junit4] 2> 3539 T1904 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:33229/solr sessionTimeout=3D30000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@1c9b2b08 [junit4:junit4] 2> 3540 T1945 oaz.ClientCnxn$SendThread.logStartConnect O= pening socket connection to server localhost.localdomain/127.0.0.1:33229. W= ill not attempt to authenticate using SASL (access denied (javax.security.a= uth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 3540 T1904 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 3540 T1945 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:33229, init= iating session [junit4:junit4] 2> 3540 T1906 oazs.NIOServerCnxnFactory.run Accepted sock= et connection from /127.0.0.1:41308 [junit4:junit4] 2> 3540 T1906 oazs.ZooKeeperServer.processConnectRequest = Client attempting to establish new session at /127.0.0.1:41308 [junit4:junit4] 2> 3541 T1908 oazs.ZooKeeperServer.finishSessionInit Esta= blished session 0x13d39504a780006 with negotiated timeout 20000 for client = /127.0.0.1:41308 [junit4:junit4] 2> 3541 T1945 oaz.ClientCnxn$SendThread.onConnected Sessi= on establishment complete on server localhost.localdomain/127.0.0.1:33229, = sessionid =3D 0x13d39504a780006, negotiated timeout =3D 20000 [junit4:junit4] 2> 3541 T1946 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@1c9b2b08 name:ZooKeeperConnecti= on Watcher:127.0.0.1:33229/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 3542 T1904 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 3542 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780006 type:cre= ate cxid:0x1 zxid:0x49 txntype:-1 reqpath:n/a Error Path:/solr/overseer Err= or:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 3543 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780006 type:cre= ate cxid:0x2 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Err= or:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 3544 T1904 oascc.ZkStateReader.createClusterStateWatch= ersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 4546 T1904 oasc.ZkController.createEphemeralLiveNode R= egister node as live in ZooKeeper:/live_nodes/127.0.0.1:55383_um_xr [junit4:junit4] 2> 4547 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780006 type:del= ete cxid:0x9 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/1= 27.0.0.1:55383_um_xr Error:KeeperErrorCode =3D NoNode for /solr/live_nodes/= 127.0.0.1:55383_um_xr [junit4:junit4] 2> 4548 T1904 oascc.SolrZkClient.makePath makePath: /live= _nodes/127.0.0.1:55383_um_xr [junit4:junit4] 2> 4549 T1926 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/= live_nodes, has occurred - updating... (live nodes size: 1) [junit4:junit4] 2> 4550 T1933 oascc.ZkStateReader$3.process Updating live= nodes... (2) [junit4:junit4] 2> 4550 T1946 oascc.ZkStateReader$3.process Updating live= nodes... (2) [junit4:junit4] 2> 4550 T1926 oascc.ZkStateReader$3.process Updating live= nodes... (2) [junit4:junit4] 2> 4554 T1947 oasc.CoreContainer.create Creating SolrCore= 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-tr= unk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyS= afeLeaderTest-jetty1-1362466199488/collection1 [junit4:junit4] 2> 4555 T1947 oasc.ZkController.createCollectionZkNode Ch= eck for collection zkNode:collection1 [junit4:junit4] 2> 4555 T1947 oasc.ZkController.createCollectionZkNode Co= llection zkNode exists [junit4:junit4] 2> 4555 T1947 oasc.ZkController.readConfigName Load colle= ction config from:/collections/collection1 [junit4:junit4] 2> 4556 T1947 oasc.SolrResourceLoader. new SolrReso= urceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Lin= ux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLead= erTest-jetty1-1362466199488/collection1/' [junit4:junit4] 2> 4557 T1947 oasc.SolrResourceLoader.replaceClassLoader = Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-13= 62466199488/collection1/lib/README' to classloader [junit4:junit4] 2> 4557 T1947 oasc.SolrResourceLoader.replaceClassLoader = Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-13= 62466199488/collection1/lib/classes/' to classloader [junit4:junit4] 2> 4587 T1947 oasc.SolrConfig. Using Lucene MatchVe= rsion: LUCENE_50 [junit4:junit4] 2> 4632 T1947 oasc.SolrConfig. Loaded SolrConfig: s= olrconfig.xml [junit4:junit4] 2> 4633 T1947 oass.IndexSchema.readSchema Reading Solr Sc= hema [junit4:junit4] 2> 4639 T1947 oass.IndexSchema.readSchema Schema name=3Dt= est [junit4:junit4] 2> 4896 T1927 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 4897 T1927 oasc.Overseer$ClusterStateUpdater.updateSta= te Update state numShards=3D1 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"1", [junit4:junit4] 2> =09 "shard":"shard1", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"control_collection", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:53417_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:53417/um_xr"} [junit4:junit4] 2> 4899 T1946 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 4899 T1933 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 4900 T1926 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 5100 T1947 oass.OpenExchangeRatesOrgProvider.init Init= ialized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 5106 T1947 oass.IndexSchema.readSchema default search = field in schema is text [junit4:junit4] 2> 5110 T1947 oass.IndexSchema.readSchema unique key fiel= d: id [junit4:junit4] 2> 5122 T1947 oass.FileExchangeRateProvider.reload Reload= ing exchange rates from file currency.xml [junit4:junit4] 2> 5125 T1947 oass.FileExchangeRateProvider.reload Reload= ing exchange rates from file currency.xml [junit4:junit4] 2> 5128 T1947 oass.OpenExchangeRatesOrgProvider.reload Re= loading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 5130 T1947 oass.OpenExchangeRatesOrgProvider$OpenExcha= ngeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 5130 T1947 oass.OpenExchangeRatesOrgProvider$OpenExcha= ngeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 5130 T1947 oass.OpenExchangeRatesOrgProvider.reload Re= loading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 5131 T1947 oass.OpenExchangeRatesOrgProvider$OpenExcha= ngeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 5131 T1947 oass.OpenExchangeRatesOrgProvider$OpenExcha= ngeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 5131 T1947 oasc.SolrCore. [collection1] Opening = new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bui= ld/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jett= y1-1362466199488/collection1/, dataDir=3D./org.apache.solr.cloud.ChaosMonke= ySafeLeaderTest-1362466196085/jetty1/ [junit4:junit4] 2> 5131 T1947 oasc.JmxMonitoredMap. JMX monitoring = is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMB= eanServer@1c1fa491 [junit4:junit4] 2> 5132 T1947 oasc.SolrCore.initDirectoryFactory solr.Sta= ndardDirectoryFactory [junit4:junit4] 2> 5132 T1947 oasc.CachingDirectoryFactory.get return new= directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/buil= d/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624= 66196085/jetty1 forceNew:false [junit4:junit4] 2> 5132 T1947 oasc.CachingDirectoryFactory.close Releasin= g directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/s= olr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624661= 96085/jetty1 [junit4:junit4] 2> 5133 T1947 oasc.SolrCore.getNewIndexDir New index dire= ctory detected: old=3Dnull new=3D./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-1362466196085/jetty1/index/ [junit4:junit4] 2> 5133 T1947 oasc.SolrCore.initIndex WARNING [collection= 1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-= 1362466196085/jetty1/index' doesn't exist. Creating new index... [junit4:junit4] 2> 5133 T1947 oasc.CachingDirectoryFactory.get return new= directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/buil= d/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624= 66196085/jetty1/index forceNew:false [junit4:junit4] 2> 5136 T1947 oasc.SolrDeletionPolicy.onCommit SolrDeleti= onPolicy.onCommit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty1/index,segFN=3Dsegments_1,generation= =3D1,filenames=3D[segments_1] [junit4:junit4] 2> 5137 T1947 oasc.SolrDeletionPolicy.updateCommits newes= t commit =3D 1[segments_1] [junit4:junit4] 2> 5137 T1947 oasc.CachingDirectoryFactory.close Releasin= g directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/s= olr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624661= 96085/jetty1/index [junit4:junit4] 2> 5138 T1947 oasup.UpdateRequestProcessorChain.init inse= rting DistributedUpdateProcessorFactory into updateRequestProcessorChain "d= edupe" [junit4:junit4] 2> 5138 T1947 oasup.UpdateRequestProcessorChain.init inse= rting DistributedUpdateProcessorFactory into updateRequestProcessorChain "s= tored_sig" [junit4:junit4] 2> 5138 T1947 oasup.UpdateRequestProcessorChain.init inse= rting DistributedUpdateProcessorFactory into updateRequestProcessorChain "d= istrib-dup-test-chain-implicit" [junit4:junit4] 2> 5139 T1947 oasc.RequestHandlers.initHandlersFromConfig= created standard: solr.StandardRequestHandler [junit4:junit4] 2> 5139 T1947 oasc.RequestHandlers.initHandlersFromConfig= created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 5140 T1947 oasc.RequestHandlers.initHandlersFromConfig= adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 5140 T1947 oasc.RequestHandlers.initHandlersFromConfig= created /replication: solr.ReplicationHandler [junit4:junit4] 2> 5140 T1947 oasc.RequestHandlers.initHandlersFromConfig= created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 5140 T1947 oasc.RequestHandlers.initHandlersFromConfig= created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 5142 T1947 oashl.XMLLoader.init xsltCacheLifetimeSecon= ds=3D60 [junit4:junit4] 2> 5145 T1947 oasc.CachingDirectoryFactory.close Releasin= g directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/s= olr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624661= 96085/jetty1 [junit4:junit4] 2> 5146 T1947 oass.SolrIndexSearcher. Opening Searc= her@1c76f0c4 main [junit4:junit4] 2> 5146 T1947 oasu.CommitTracker. Hard AutoCommit: = disabled [junit4:junit4] 2> 5146 T1947 oasu.CommitTracker. Soft AutoCommit: = disabled [junit4:junit4] 2> 5152 T1948 oasc.SolrCore.registerSearcher [collection1= ] Registered new searcher Searcher@1c76f0c4 main{StandardDirectoryReader(se= gments_1:1)} [junit4:junit4] 2> 5154 T1947 oasc.ZkController.publish publishing core= =3Dcollection1 state=3Ddown [junit4:junit4] 2> 5155 T1947 oasc.ZkController.publish numShards not fou= nd on descriptor - reading it from system property [junit4:junit4] 2> 6403 T1927 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 6404 T1927 oasc.Overseer$ClusterStateUpdater.updateSta= te Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":null, [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"down", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:55383_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:55383/um_xr"} [junit4:junit4] 2> 6404 T1927 oasc.Overseer$ClusterStateUpdater.createCol= lection Create collection collection1 with numShards 3 [junit4:junit4] 2> 6404 T1927 oasc.Overseer$ClusterStateUpdater.updateSta= te Assigning new node to shard shard=3Dshard1 [junit4:junit4] 2> 6408 T1926 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 6408 T1933 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 6408 T1946 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 7156 T1947 oasc.CoreContainer.registerCore registering= core: collection1 [junit4:junit4] 2> 7157 T1947 oasc.ZkController.register Register replica= - core:collection1 address:http://127.0.0.1:55383/um_xr collection:collect= ion1 shard:shard1 [junit4:junit4] 2> 7157 T1947 oascc.SolrZkClient.makePath makePath: /coll= ections/collection1/leader_elect/shard1/election [junit4:junit4] 2> 7163 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780006 type:del= ete cxid:0x46 zxid:0x5b txntype:-1 reqpath:n/a Error Path:/solr/collections= /collection1/leaders Error:KeeperErrorCode =3D NoNode for /solr/collections= /collection1/leaders [junit4:junit4] 2> 7163 T1947 oasc.ShardLeaderElectionContext.runLeaderPr= ocess Running the leader process. [junit4:junit4] 2> 7164 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780006 type:cre= ate cxid:0x47 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 7165 T1947 oasc.ShardLeaderElectionContext.waitForRepl= icasToComeUp Enough replicas found to continue. [junit4:junit4] 2> 7165 T1947 oasc.ShardLeaderElectionContext.runLeaderPr= ocess I may be the new leader - try and sync [junit4:junit4] 2> 7165 T1947 oasc.SyncStrategy.sync Sync replicas to htt= p://127.0.0.1:55383/um_xr/collection1/ [junit4:junit4] 2> 7166 T1947 oasc.SyncStrategy.syncReplicas Sync Success= - now sync replicas to me [junit4:junit4] 2> 7166 T1947 oasc.SyncStrategy.syncToMe http://127.0.0.1= :55383/um_xr/collection1/ has no replicas [junit4:junit4] 2> 7166 T1947 oasc.ShardLeaderElectionContext.runLeaderPr= ocess I am the new leader: http://127.0.0.1:55383/um_xr/collection1/ [junit4:junit4] 2> 7166 T1947 oascc.SolrZkClient.makePath makePath: /coll= ections/collection1/leaders/shard1 [junit4:junit4] 2> 7170 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780006 type:cre= ate cxid:0x51 zxid:0x60 txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 7912 T1927 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 8026 T1926 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 8026 T1933 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 8026 T1946 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 8073 T1947 oasc.ZkController.register We are http://12= 7.0.0.1:55383/um_xr/collection1/ and leader is http://127.0.0.1:55383/um_xr= /collection1/ [junit4:junit4] 2> 8074 T1947 oasc.ZkController.register No LogReplay nee= ded for core=3Dcollection1 baseURL=3Dhttp://127.0.0.1:55383/um_xr [junit4:junit4] 2> 8074 T1947 oasc.ZkController.checkRecovery I am the le= ader, no recovery necessary [junit4:junit4] 2> 8074 T1947 oasc.ZkController.publish publishing core= =3Dcollection1 state=3Dactive [junit4:junit4] 2> 8074 T1947 oasc.ZkController.publish numShards not fou= nd on descriptor - reading it from system property [junit4:junit4] 2> 8076 T1947 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 8077 T1904 oass.SolrDispatchFilter.init user.dir=3D/mn= t/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J= 1 [junit4:junit4] 2> 8077 T1904 oass.SolrDispatchFilter.init SolrDispatchFi= lter.init() done [junit4:junit4] 2> 8078 T1904 oascsi.HttpClientUtil.createClient Creating= new http client, config:maxConnections=3D128&maxConnectionsPerHost=3D32&fo= llowRedirects=3Dfalse [junit4:junit4] 2> 8141 T1904 oejs.Server.doStart jetty-8.1.8.v20121106 [junit4:junit4] 2> 8143 T1904 oejs.AbstractConnector.doStart Started Sele= ctChannelConnector@127.0.0.1:35943 [junit4:junit4] 2> 8144 T1904 oass.SolrDispatchFilter.init SolrDispatchFi= lter.init() [junit4:junit4] 2> 8144 T1904 oasc.SolrResourceLoader.locateSolrHome JNDI= not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 8144 T1904 oasc.SolrResourceLoader.locateSolrHome usin= g system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-tr= unk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyS= afeLeaderTest-jetty2-1362466204145 [junit4:junit4] 2> 8145 T1904 oasc.CoreContainer$Initializer.initialize l= ooking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-L= inux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLe= aderTest-jetty2-1362466204145/solr.xml [junit4:junit4] 2> 8145 T1904 oasc.CoreContainer. New CoreContainer= 479639511 [junit4:junit4] 2> 8145 T1904 oasc.CoreContainer.load Loading CoreContain= er using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/sol= r/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest= -jetty2-1362466204145/' [junit4:junit4] 2> 8146 T1904 oasc.SolrResourceLoader. new SolrReso= urceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Lin= ux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLead= erTest-jetty2-1362466204145/' [junit4:junit4] 2> 8162 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting socketTimeout to: 120000 [junit4:junit4] 2> 8163 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting urlScheme to: http:// [junit4:junit4] 2> 8163 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting connTimeout to: 15000 [junit4:junit4] 2> 8163 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 8164 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting corePoolSize to: 0 [junit4:junit4] 2> 8164 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 8164 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting maxThreadIdleTime to: 5 [junit4:junit4] 2> 8164 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting sizeOfQueue to: -1 [junit4:junit4] 2> 8165 T1904 oashc.HttpShardHandlerFactory.getParameter = Setting fairnessPolicy to: false [junit4:junit4] 2> 8165 T1904 oascsi.HttpClientUtil.createClient Creating= new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000&= socketTimeout=3D120000&connTimeout=3D15000&retry=3Dfalse [junit4:junit4] 2> 8172 T1904 oasc.CoreContainer.load Registering Log Lis= tener [junit4:junit4] 2> 8185 T1904 oasc.CoreContainer.initZooKeeper Zookeeper = client=3D127.0.0.1:33229/solr [junit4:junit4] 2> 8186 T1904 oasc.ZkController.checkChrootPath zkHost in= cludes chroot [junit4:junit4] 2> 8186 T1904 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:33229 sessionTimeout=3D60000 watcher=3Dor= g.apache.solr.common.cloud.ConnectionManager@1c9b33bf [junit4:junit4] 2> 8187 T1904 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 8187 T1959 oaz.ClientCnxn$SendThread.logStartConnect O= pening socket connection to server localhost.localdomain/127.0.0.1:33229. W= ill not attempt to authenticate using SASL (access denied (javax.security.a= uth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 8187 T1959 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:33229, init= iating session [junit4:junit4] 2> 8188 T1906 oazs.NIOServerCnxnFactory.run Accepted sock= et connection from /127.0.0.1:41309 [junit4:junit4] 2> 8188 T1906 oazs.ZooKeeperServer.processConnectRequest = Client attempting to establish new session at /127.0.0.1:41309 [junit4:junit4] 2> 8188 T1908 oazs.ZooKeeperServer.finishSessionInit Esta= blished session 0x13d39504a780007 with negotiated timeout 20000 for client = /127.0.0.1:41309 [junit4:junit4] 2> 8188 T1959 oaz.ClientCnxn$SendThread.onConnected Sessi= on establishment complete on server localhost.localdomain/127.0.0.1:33229, = sessionid =3D 0x13d39504a780007, negotiated timeout =3D 20000 [junit4:junit4] 2> 8189 T1960 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@1c9b33bf name:ZooKeeperConnecti= on Watcher:127.0.0.1:33229 got event WatchedEvent state:SyncConnected type:= None path:null path:null type:None [junit4:junit4] 2> 8189 T1904 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 8190 T1909 oazs.PrepRequestProcessor.pRequest2Txn Proc= essed session termination for sessionid: 0x13d39504a780007 [junit4:junit4] 2> 8191 T1906 oazs.NIOServerCnxn.closeSock Closed socket = connection for client /127.0.0.1:41309 which had sessionid 0x13d39504a78000= 7 [junit4:junit4] 2> 8191 T1960 oaz.ClientCnxn$EventThread.run EventThread = shut down [junit4:junit4] 2> 8191 T1904 oaz.ZooKeeper.close Session: 0x13d39504a780= 007 closed [junit4:junit4] 2> 8191 T1904 oascsi.HttpClientUtil.createClient Creating= new http client, config:maxConnections=3D500&maxConnectionsPerHost=3D16&so= cketTimeout=3D120000&connTimeout=3D15000 [junit4:junit4] 2> 8194 T1904 oaz.ZooKeeper. Initiating client conn= ection, connectString=3D127.0.0.1:33229/solr sessionTimeout=3D30000 watcher= =3Dorg.apache.solr.common.cloud.ConnectionManager@1c9c31c3 [junit4:junit4] 2> 8195 T1961 oaz.ClientCnxn$SendThread.logStartConnect O= pening socket connection to server localhost.localdomain/127.0.0.1:33229. W= ill not attempt to authenticate using SASL (access denied (javax.security.a= uth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 8195 T1904 oascc.ConnectionManager.waitForConnected Wa= iting for client to connect to ZooKeeper [junit4:junit4] 2> 8195 T1961 oaz.ClientCnxn$SendThread.primeConnection S= ocket connection established to localhost.localdomain/127.0.0.1:33229, init= iating session [junit4:junit4] 2> 8195 T1906 oazs.NIOServerCnxnFactory.run Accepted sock= et connection from /127.0.0.1:41310 [junit4:junit4] 2> 8195 T1906 oazs.ZooKeeperServer.processConnectRequest = Client attempting to establish new session at /127.0.0.1:41310 [junit4:junit4] 2> 8196 T1908 oazs.ZooKeeperServer.finishSessionInit Esta= blished session 0x13d39504a780008 with negotiated timeout 20000 for client = /127.0.0.1:41310 [junit4:junit4] 2> 8196 T1961 oaz.ClientCnxn$SendThread.onConnected Sessi= on establishment complete on server localhost.localdomain/127.0.0.1:33229, = sessionid =3D 0x13d39504a780008, negotiated timeout =3D 20000 [junit4:junit4] 2> 8197 T1962 oascc.ConnectionManager.process Watcher org= .apache.solr.common.cloud.ConnectionManager@1c9c31c3 name:ZooKeeperConnecti= on Watcher:127.0.0.1:33229/solr got event WatchedEvent state:SyncConnected = type:None path:null path:null type:None [junit4:junit4] 2> 8197 T1904 oascc.ConnectionManager.waitForConnected Cl= ient is connected to ZooKeeper [junit4:junit4] 2> 8197 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780008 type:cre= ate cxid:0x1 zxid:0x6d txntype:-1 reqpath:n/a Error Path:/solr/overseer Err= or:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 8198 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780008 type:cre= ate cxid:0x2 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Err= or:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 8199 T1904 oascc.ZkStateReader.createClusterStateWatch= ersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 9202 T1904 oasc.ZkController.createEphemeralLiveNode R= egister node as live in ZooKeeper:/live_nodes/127.0.0.1:35943_um_xr [junit4:junit4] 2> 9203 T1909 oazs.PrepRequestProcessor.pRequest Got user= -level KeeperException when processing sessionid:0x13d39504a780008 type:del= ete cxid:0x9 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/1= 27.0.0.1:35943_um_xr Error:KeeperErrorCode =3D NoNode for /solr/live_nodes/= 127.0.0.1:35943_um_xr [junit4:junit4] 2> 9204 T1904 oascc.SolrZkClient.makePath makePath: /live= _nodes/127.0.0.1:35943_um_xr [junit4:junit4] 2> 9208 T1926 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/= live_nodes, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 9208 T1946 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/= live_nodes, has occurred - updating... (live nodes size: 2) [junit4:junit4] 2> 9209 T1933 oascc.ZkStateReader$3.process Updating live= nodes... (3) [junit4:junit4] 2> 9209 T1933 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/= live_nodes, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 9209 T1962 oascc.ZkStateReader$3.process Updating live= nodes... (3) [junit4:junit4] 2> 9211 T1926 oascc.ZkStateReader$3.process Updating live= nodes... (3) [junit4:junit4] 2> 9211 T1946 oascc.ZkStateReader$3.process Updating live= nodes... (3) [junit4:junit4] 2> 9219 T1963 oasc.CoreContainer.create Creating SolrCore= 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-tr= unk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyS= afeLeaderTest-jetty2-1362466204145/collection1 [junit4:junit4] 2> 9220 T1963 oasc.ZkController.createCollectionZkNode Ch= eck for collection zkNode:collection1 [junit4:junit4] 2> 9221 T1963 oasc.ZkController.createCollectionZkNode Co= llection zkNode exists [junit4:junit4] 2> 9221 T1963 oasc.ZkController.readConfigName Load colle= ction config from:/collections/collection1 [junit4:junit4] 2> 9223 T1963 oasc.SolrResourceLoader. new SolrReso= urceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Lin= ux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLead= erTest-jetty2-1362466204145/collection1/' [junit4:junit4] 2> 9224 T1963 oasc.SolrResourceLoader.replaceClassLoader = Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-13= 62466204145/collection1/lib/README' to classloader [junit4:junit4] 2> 9225 T1963 oasc.SolrResourceLoader.replaceClassLoader = Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-13= 62466204145/collection1/lib/classes/' to classloader [junit4:junit4] 2> 9254 T1963 oasc.SolrConfig. Using Lucene MatchVe= rsion: LUCENE_50 [junit4:junit4] 2> 9293 T1963 oasc.SolrConfig. Loaded SolrConfig: s= olrconfig.xml [junit4:junit4] 2> 9294 T1963 oass.IndexSchema.readSchema Reading Solr Sc= hema [junit4:junit4] 2> 9299 T1963 oass.IndexSchema.readSchema Schema name=3Dt= est [junit4:junit4] 2> 9532 T1927 oascc.ZkStateReader.updateClusterState Upda= ting cloud state from ZooKeeper...=20 [junit4:junit4] 2> 9533 T1927 oasc.Overseer$ClusterStateUpdater.updateSta= te Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":"shard1", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:55383_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:55383/um_xr"} [junit4:junit4] 2> 9535 T1926 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 9535 T1946 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 9535 T1962 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 9535 T1933 oascc.ZkStateReader$2.process A cluster sta= te change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clus= terstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 9750 T1963 oass.OpenExchangeRatesOrgProvider.init Init= ialized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 9757 T1963 oass.IndexSchema.readSchema default search = field in schema is text [junit4:junit4] 2> 9760 T1963 oass.IndexSchema.readSchema unique key fiel= d: id [junit4:junit4] 2> 9772 T1963 oass.FileExchangeRateProvider.reload Reload= ing exchange rates from file currency.xml [junit4:junit4] 2> 9775 T1963 oass.FileExchangeRateProvider.reload Reload= ing exchange rates from file currency.xml [junit4:junit4] 2> 9778 T1963 oass.OpenExchangeRatesOrgProvider.reload Re= loading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 9779 T1963 oass.OpenExchangeRatesOrgProvider$OpenExcha= ngeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 9779 T1963 oass.OpenExchangeRatesOrgProvider$OpenExcha= ngeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 9779 T1963 oass.OpenExchangeRatesOrgProvider.reload Re= loading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 9780 T1963 oass.OpenExchangeRatesOrgProvider$OpenExcha= ngeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 9781 T1963 oass.OpenExchangeRatesOrgProvider$OpenExcha= ngeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 9781 T1963 oasc.SolrCore. [collection1] Opening = new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bui= ld/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jett= y2-1362466204145/collection1/, dataDir=3D./org.apache.solr.cloud.ChaosMonke= ySafeLeaderTest-1362466196085/jetty2/ [junit4:junit4] 2> 9781 T1963 oasc.JmxMonitoredMap. JMX monitoring = is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMB= eanServer@1c1fa491 [junit4:junit4] 2> 9781 T1963 oasc.SolrCore.initDirectoryFactory solr.Sta= ndardDirectoryFactory [junit4:junit4] 2> 9782 T1963 oasc.CachingDirectoryFactory.get return new= directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/buil= d/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624= 66196085/jetty2 forceNew:false [junit4:junit4] 2> 9782 T1963 oasc.CachingDirectoryFactory.close Releasin= g directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/s= olr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624661= 96085/jetty2 [junit4:junit4] 2> 9782 T1963 oasc.SolrCore.getNewIndexDir New index dire= ctory detected: old=3Dnull new=3D./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-1362466196085/jetty2/index/ [junit4:junit4] 2> 9783 T1963 oasc.SolrCore.initIndex WARNING [collection= 1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-= 1362466196085/jetty2/index' doesn't exist. Creating new index... [junit4:junit4] 2> 9783 T1963 oasc.CachingDirectoryFactory.get return new= directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/buil= d/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624= 66196085/jetty2/index forceNew:false [junit4:junit4] 2> 9786 T1963 oasc.SolrDeletionPolicy.onCommit SolrDeleti= onPolicy.onCommit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty2/index,segFN=3Dsegments_1,generation= =3D1,filenames=3D[segments_1] [junit4:junit4] 2> 9786 T1963 oasc.SolrDeletionPolicy.updateCommits newes= t commit =3D 1[segments_1] [junit4:junit4] 2> 9786 T1963 oasc.CachingDirectoryFactory.close Releasin= g directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/s= olr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624661= 96085/jetty2/index [junit4:junit4] 2> 9787 T1963 oasup.UpdateRequestProcessorChain.init inse= rting DistributedUpdateProcessorFactory into updateRequestProcessorChain "d= edupe" [junit4:junit4] 2> 9788 T1963 oasup.UpdateRequestProcessorChain.init inse= rting DistributedUpdateProcessorFactory into updateRequestProcessorChain "s= tored_sig" [junit4:junit4] 2> 9788 T1963 oasup.UpdateRequestProcessorChain.init inse= rting DistributedUpdateProcessorFactory into updateRequestProcessorChain "d= istrib-dup-test-chain-implicit" [junit4:junit4] 2> 9788 T1963 oasc.RequestHandlers.initHandlersFromConfig= created standard: solr.StandardRequestHandler [junit4:junit4] 2> 9789 T1963 oasc.RequestHandlers.initHandlersFromConfig= created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 9789 T1963 oasc.RequestHandlers.initHandlersFromConfig= adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 9789 T1963 oasc.RequestHandlers.initHandlersFromConfig= created /replication: solr.ReplicationHandler [junit4:junit4] 2> 9790 T1963 oasc.RequestHandlers.initHandlersFromConfig= created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 9790 T1963 oasc.RequestHandlers.initHandlersFromConfig= created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 9792 T1963 oashl.XMLLoader.init xsltCacheLifetimeSecon= ds=3D60 [junit4:junit4] 2> 9795 T1963 oasc.CachingDirectoryFactory.close Releasin= g directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/s= olr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-13624661= 96085/jetty2 [junit4:junit4] 2> 9795 T1963 oass.SolrIndexSearcher. Opening Searc= her@1c8276a2 main [junit4:junit4] 2> 9796 T1963 oasu.CommitTracker. Hard AutoCommit: = disabled [junit4:junit4] 2> 9796 T1963 oasu.CommitTracker. Soft AutoCommit: = disabled [junit4:junit4] 2> 9803 T1964 oasc.SolrCore.registerSearcher [collection1= ] Registered new searcher Searcher@1c8276a2 main{StandardDirectoryReader(se= gments_1:1)} [junit4:junit4] 2> 9806 T1963 oasc.ZkController.publish publishing core= =3Dcollection1 state=3Ddown [junit4:junit4] 2> 9807 T1963 oasc.ZkController.publish numShards not fou= nd on descriptor - reading it from system property [junit4:junit4] 2> 11041 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 11042 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":null, [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"down", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:35943_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:35943/um_xr"} [junit4:junit4] 2> 11043 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Collection already exists with numShards=3D3 [junit4:junit4] 2> 11043 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Assigning new node to shard shard=3Dshard2 [junit4:junit4] 2> 11049 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 11049 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 11049 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 11049 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 11810 T1963 oasc.CoreContainer.registerCore registerin= g core: collection1 [junit4:junit4] 2> 11810 T1963 oasc.ZkController.register Register replic= a - core:collection1 address:http://127.0.0.1:35943/um_xr collection:collec= tion1 shard:shard2 [junit4:junit4] 2> 11811 T1963 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leader_elect/shard2/election [junit4:junit4] 2> 11822 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a780008 type:de= lete cxid:0x45 zxid:0x7e txntype:-1 reqpath:n/a Error Path:/solr/collection= s/collection1/leaders/shard2 Error:KeeperErrorCode =3D NoNode for /solr/col= lections/collection1/leaders/shard2 [junit4:junit4] 2> 11823 T1963 oasc.ShardLeaderElectionContext.runLeaderP= rocess Running the leader process. [junit4:junit4] 2> 11824 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a780008 type:cr= eate cxid:0x46 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/overseer E= rror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 11827 T1963 oasc.ShardLeaderElectionContext.waitForRep= licasToComeUp Enough replicas found to continue. [junit4:junit4] 2> 11827 T1963 oasc.ShardLeaderElectionContext.runLeaderP= rocess I may be the new leader - try and sync [junit4:junit4] 2> 11827 T1963 oasc.SyncStrategy.sync Sync replicas to ht= tp://127.0.0.1:35943/um_xr/collection1/ [junit4:junit4] 2> 11828 T1963 oasc.SyncStrategy.syncReplicas Sync Succes= s - now sync replicas to me [junit4:junit4] 2> 11828 T1963 oasc.SyncStrategy.syncToMe http://127.0.0.= 1:35943/um_xr/collection1/ has no replicas [junit4:junit4] 2> 11829 T1963 oasc.ShardLeaderElectionContext.runLeaderP= rocess I am the new leader: http://127.0.0.1:35943/um_xr/collection1/ [junit4:junit4] 2> 11829 T1963 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leaders/shard2 [junit4:junit4] 2> 11835 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a780008 type:cr= eate cxid:0x4f zxid:0x82 txntype:-1 reqpath:n/a Error Path:/solr/overseer E= rror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 12556 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 12568 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 12568 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 12568 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 12568 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 12590 T1963 oasc.ZkController.register We are http://1= 27.0.0.1:35943/um_xr/collection1/ and leader is http://127.0.0.1:35943/um_x= r/collection1/ [junit4:junit4] 2> 12591 T1963 oasc.ZkController.register No LogReplay ne= eded for core=3Dcollection1 baseURL=3Dhttp://127.0.0.1:35943/um_xr [junit4:junit4] 2> 12591 T1963 oasc.ZkController.checkRecovery I am the l= eader, no recovery necessary [junit4:junit4] 2> 12591 T1963 oasc.ZkController.publish publishing core= =3Dcollection1 state=3Dactive [junit4:junit4] 2> 12592 T1963 oasc.ZkController.publish numShards not fo= und on descriptor - reading it from system property [junit4:junit4] 2> 12594 T1963 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 12596 T1904 oass.SolrDispatchFilter.init user.dir=3D/m= nt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/= J1 [junit4:junit4] 2> 12596 T1904 oass.SolrDispatchFilter.init SolrDispatchF= ilter.init() done [junit4:junit4] 2> 12597 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnections=3D128&maxConnectionsPerHost=3D32&f= ollowRedirects=3Dfalse [junit4:junit4] 2> 12660 T1904 oejs.Server.doStart jetty-8.1.8.v20121106 [junit4:junit4] 2> 12663 T1904 oejs.AbstractConnector.doStart Started Sel= ectChannelConnector@127.0.0.1:35931 [junit4:junit4] 2> 12663 T1904 oass.SolrDispatchFilter.init SolrDispatchF= ilter.init() [junit4:junit4] 2> 12663 T1904 oasc.SolrResourceLoader.locateSolrHome JND= I not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 12664 T1904 oasc.SolrResourceLoader.locateSolrHome usi= ng system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-t= runk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkey= SafeLeaderTest-jetty3-1362466208668 [junit4:junit4] 2> 12664 T1904 oasc.CoreContainer$Initializer.initialize = looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-= Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeL= eaderTest-jetty3-1362466208668/solr.xml [junit4:junit4] 2> 12664 T1904 oasc.CoreContainer. New CoreContaine= r 480735345 [junit4:junit4] 2> 12665 T1904 oasc.CoreContainer.load Loading CoreContai= ner using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/so= lr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTes= t-jetty3-1362466208668/' [junit4:junit4] 2> 12665 T1904 oasc.SolrResourceLoader. new SolrRes= ourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-jetty3-1362466208668/' [junit4:junit4] 2> 12680 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting socketTimeout to: 120000 [junit4:junit4] 2> 12681 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting urlScheme to: http:// [junit4:junit4] 2> 12681 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting connTimeout to: 15000 [junit4:junit4] 2> 12681 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 12681 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting corePoolSize to: 0 [junit4:junit4] 2> 12682 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 12682 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maxThreadIdleTime to: 5 [junit4:junit4] 2> 12682 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting sizeOfQueue to: -1 [junit4:junit4] 2> 12682 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting fairnessPolicy to: false [junit4:junit4] 2> 12683 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000= &socketTimeout=3D120000&connTimeout=3D15000&retry=3Dfalse [junit4:junit4] 2> 12689 T1904 oasc.CoreContainer.load Registering Log Li= stener [junit4:junit4] 2> 12702 T1904 oasc.CoreContainer.initZooKeeper Zookeeper= client=3D127.0.0.1:33229/solr [junit4:junit4] 2> 12702 T1904 oasc.ZkController.checkChrootPath zkHost i= ncludes chroot [junit4:junit4] 2> 12703 T1904 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:33229 sessionTimeout=3D60000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@1c8a3360 [junit4:junit4] 2> 12704 T1975 oaz.ClientCnxn$SendThread.logStartConnect = Opening socket connection to server localhost.localdomain/127.0.0.1:33229. = Will not attempt to authenticate using SASL (access denied (javax.security.= auth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 12704 T1904 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 12704 T1975 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:33229, ini= tiating session [junit4:junit4] 2> 12704 T1906 oazs.NIOServerCnxnFactory.run Accepted soc= ket connection from /127.0.0.1:41336 [junit4:junit4] 2> 12704 T1906 oazs.ZooKeeperServer.processConnectRequest= Client attempting to establish new session at /127.0.0.1:41336 [junit4:junit4] 2> 12705 T1908 oazs.ZooKeeperServer.finishSessionInit Est= ablished session 0x13d39504a780009 with negotiated timeout 20000 for client= /127.0.0.1:41336 [junit4:junit4] 2> 12705 T1975 oaz.ClientCnxn$SendThread.onConnected Sess= ion establishment complete on server localhost.localdomain/127.0.0.1:33229,= sessionid =3D 0x13d39504a780009, negotiated timeout =3D 20000 [junit4:junit4] 2> 12705 T1976 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1c8a3360 name:ZooKeeperConnect= ion Watcher:127.0.0.1:33229 got event WatchedEvent state:SyncConnected type= :None path:null path:null type:None [junit4:junit4] 2> 12706 T1904 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 12707 T1909 oazs.PrepRequestProcessor.pRequest2Txn Pro= cessed session termination for sessionid: 0x13d39504a780009 [junit4:junit4] 2> 12708 T1976 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 12708 T1904 oaz.ZooKeeper.close Session: 0x13d39504a78= 0009 closed [junit4:junit4] 2> 12708 T1906 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:41336 which had sessionid 0x13d39504a7800= 09 [junit4:junit4] 2> 12708 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnections=3D500&maxConnectionsPerHost=3D16&s= ocketTimeout=3D120000&connTimeout=3D15000 [junit4:junit4] 2> 12711 T1904 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:33229/solr sessionTimeout=3D30000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1c8bcdfe [junit4:junit4] 2> 12712 T1977 oaz.ClientCnxn$SendThread.logStartConnect = Opening socket connection to server localhost.localdomain/127.0.0.1:33229. = Will not attempt to authenticate using SASL (access denied (javax.security.= auth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 12712 T1904 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 12712 T1977 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:33229, ini= tiating session [junit4:junit4] 2> 12712 T1906 oazs.NIOServerCnxnFactory.run Accepted soc= ket connection from /127.0.0.1:41337 [junit4:junit4] 2> 12713 T1906 oazs.ZooKeeperServer.processConnectRequest= Client attempting to establish new session at /127.0.0.1:41337 [junit4:junit4] 2> 12713 T1908 oazs.ZooKeeperServer.finishSessionInit Est= ablished session 0x13d39504a78000a with negotiated timeout 20000 for client= /127.0.0.1:41337 [junit4:junit4] 2> 12713 T1977 oaz.ClientCnxn$SendThread.onConnected Sess= ion establishment complete on server localhost.localdomain/127.0.0.1:33229,= sessionid =3D 0x13d39504a78000a, negotiated timeout =3D 20000 [junit4:junit4] 2> 12714 T1978 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1c8bcdfe name:ZooKeeperConnect= ion Watcher:127.0.0.1:33229/solr got event WatchedEvent state:SyncConnected= type:None path:null path:null type:None [junit4:junit4] 2> 12714 T1904 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 12714 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000a type:cr= eate cxid:0x1 zxid:0x8f txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 12716 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000a type:cr= eate cxid:0x2 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 12718 T1904 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 13721 T1904 oasc.ZkController.createEphemeralLiveNode = Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35931_um_xr [junit4:junit4] 2> 13722 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000a type:de= lete cxid:0x9 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/= 127.0.0.1:35931_um_xr Error:KeeperErrorCode =3D NoNode for /solr/live_nodes= /127.0.0.1:35931_um_xr [junit4:junit4] 2> 13723 T1904 oascc.SolrZkClient.makePath makePath: /liv= e_nodes/127.0.0.1:35931_um_xr [junit4:junit4] 2> 13726 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 13726 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 3) [junit4:junit4] 2> 13727 T1962 oascc.ZkStateReader$3.process Updating liv= e nodes... (4) [junit4:junit4] 2> 13727 T1978 oascc.ZkStateReader$3.process Updating liv= e nodes... (4) [junit4:junit4] 2> 13727 T1933 oascc.ZkStateReader$3.process Updating liv= e nodes... (4) [junit4:junit4] 2> 13728 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 13728 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 13730 T1946 oascc.ZkStateReader$3.process Updating liv= e nodes... (4) [junit4:junit4] 2> 13730 T1926 oascc.ZkStateReader$3.process Updating liv= e nodes... (4) [junit4:junit4] 2> 13737 T1979 oasc.CoreContainer.create Creating SolrCor= e 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-t= runk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkey= SafeLeaderTest-jetty3-1362466208668/collection1 [junit4:junit4] 2> 13738 T1979 oasc.ZkController.createCollectionZkNode C= heck for collection zkNode:collection1 [junit4:junit4] 2> 13739 T1979 oasc.ZkController.createCollectionZkNode C= ollection zkNode exists [junit4:junit4] 2> 13739 T1979 oasc.ZkController.readConfigName Load coll= ection config from:/collections/collection1 [junit4:junit4] 2> 13741 T1979 oasc.SolrResourceLoader. new SolrRes= ourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-jetty3-1362466208668/collection1/' [junit4:junit4] 2> 13742 T1979 oasc.SolrResourceLoader.replaceClassLoader= Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1= 362466208668/collection1/lib/README' to classloader [junit4:junit4] 2> 13743 T1979 oasc.SolrResourceLoader.replaceClassLoader= Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1= 362466208668/collection1/lib/classes/' to classloader [junit4:junit4] 2> 13784 T1979 oasc.SolrConfig. Using Lucene MatchV= ersion: LUCENE_50 [junit4:junit4] 2> 13842 T1979 oasc.SolrConfig. Loaded SolrConfig: = solrconfig.xml [junit4:junit4] 2> 13844 T1979 oass.IndexSchema.readSchema Reading Solr S= chema [junit4:junit4] 2> 13851 T1979 oass.IndexSchema.readSchema Schema name=3D= test [junit4:junit4] 2> 14075 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 14075 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":"shard2", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:35943_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:35943/um_xr"} [junit4:junit4] 2> 14079 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 14079 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 14079 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 14079 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 14080 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 14325 T1979 oass.OpenExchangeRatesOrgProvider.init Ini= tialized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 14332 T1979 oass.IndexSchema.readSchema default search= field in schema is text [junit4:junit4] 2> 14335 T1979 oass.IndexSchema.readSchema unique key fie= ld: id [junit4:junit4] 2> 14348 T1979 oass.FileExchangeRateProvider.reload Reloa= ding exchange rates from file currency.xml [junit4:junit4] 2> 14352 T1979 oass.FileExchangeRateProvider.reload Reloa= ding exchange rates from file currency.xml [junit4:junit4] 2> 14359 T1979 oass.OpenExchangeRatesOrgProvider.reload R= eloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 14361 T1979 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 14361 T1979 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 14361 T1979 oass.OpenExchangeRatesOrgProvider.reload R= eloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 14363 T1979 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 14363 T1979 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 14364 T1979 oasc.SolrCore. [collection1] Opening= new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bu= ild/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jet= ty3-1362466208668/collection1/, dataDir=3D./org.apache.solr.cloud.ChaosMonk= eySafeLeaderTest-1362466196085/jetty3/ [junit4:junit4] 2> 14364 T1979 oasc.JmxMonitoredMap. JMX monitoring= is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxM= BeanServer@1c1fa491 [junit4:junit4] 2> 14365 T1979 oasc.SolrCore.initDirectoryFactory solr.St= andardDirectoryFactory [junit4:junit4] 2> 14366 T1979 oasc.CachingDirectoryFactory.get return ne= w directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bui= ld/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362= 466196085/jetty3 forceNew:false [junit4:junit4] 2> 14366 T1979 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty3 [junit4:junit4] 2> 14367 T1979 oasc.SolrCore.getNewIndexDir New index dir= ectory detected: old=3Dnull new=3D./org.apache.solr.cloud.ChaosMonkeySafeLe= aderTest-1362466196085/jetty3/index/ [junit4:junit4] 2> 14368 T1979 oasc.SolrCore.initIndex WARNING [collectio= n1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest= -1362466196085/jetty3/index' doesn't exist. Creating new index... [junit4:junit4] 2> 14368 T1979 oasc.CachingDirectoryFactory.get return ne= w directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bui= ld/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362= 466196085/jetty3/index forceNew:false [junit4:junit4] 2> 14373 T1979 oasc.SolrDeletionPolicy.onCommit SolrDelet= ionPolicy.onCommit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty3/index,segFN=3Dsegments_1,generation= =3D1,filenames=3D[segments_1] [junit4:junit4] 2> 14373 T1979 oasc.SolrDeletionPolicy.updateCommits newe= st commit =3D 1[segments_1] [junit4:junit4] 2> 14374 T1979 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty3/index [junit4:junit4] 2> 14376 T1979 oasup.UpdateRequestProcessorChain.init ins= erting DistributedUpdateProcessorFactory into updateRequestProcessorChain "= dedupe" [junit4:junit4] 2> 14377 T1979 oasup.UpdateRequestProcessorChain.init ins= erting DistributedUpdateProcessorFactory into updateRequestProcessorChain "= stored_sig" [junit4:junit4] 2> 14377 T1979 oasup.UpdateRequestProcessorChain.init ins= erting DistributedUpdateProcessorFactory into updateRequestProcessorChain "= distrib-dup-test-chain-implicit" [junit4:junit4] 2> 14378 T1979 oasc.RequestHandlers.initHandlersFromConfi= g created standard: solr.StandardRequestHandler [junit4:junit4] 2> 14379 T1979 oasc.RequestHandlers.initHandlersFromConfi= g created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 14380 T1979 oasc.RequestHandlers.initHandlersFromConfi= g adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 14380 T1979 oasc.RequestHandlers.initHandlersFromConfi= g created /replication: solr.ReplicationHandler [junit4:junit4] 2> 14381 T1979 oasc.RequestHandlers.initHandlersFromConfi= g created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 14382 T1979 oasc.RequestHandlers.initHandlersFromConfi= g created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 14386 T1979 oashl.XMLLoader.init xsltCacheLifetimeSeco= nds=3D60 [junit4:junit4] 2> 14391 T1979 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty3 [junit4:junit4] 2> 14393 T1979 oass.SolrIndexSearcher. Opening Sear= cher@1cadbe30 main [junit4:junit4] 2> 14393 T1979 oasu.CommitTracker. Hard AutoCommit:= disabled [junit4:junit4] 2> 14394 T1979 oasu.CommitTracker. Soft AutoCommit:= disabled [junit4:junit4] 2> 14402 T1980 oasc.SolrCore.registerSearcher [collection= 1] Registered new searcher Searcher@1cadbe30 main{StandardDirectoryReader(s= egments_1:1)} [junit4:junit4] 2> 14406 T1979 oasc.ZkController.publish publishing core= =3Dcollection1 state=3Ddown [junit4:junit4] 2> 14406 T1979 oasc.ZkController.publish numShards not fo= und on descriptor - reading it from system property [junit4:junit4] 2> 15585 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 15587 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":null, [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"down", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:35931_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:35931/um_xr"} [junit4:junit4] 2> 15587 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Collection already exists with numShards=3D3 [junit4:junit4] 2> 15588 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Assigning new node to shard shard=3Dshard3 [junit4:junit4] 2> 15593 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 15594 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 15594 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 15594 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 15594 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 16409 T1979 oasc.CoreContainer.registerCore registerin= g core: collection1 [junit4:junit4] 2> 16409 T1979 oasc.ZkController.register Register replic= a - core:collection1 address:http://127.0.0.1:35931/um_xr collection:collec= tion1 shard:shard3 [junit4:junit4] 2> 16411 T1979 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leader_elect/shard3/election [junit4:junit4] 2> 16420 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000a type:de= lete cxid:0x45 zxid:0xa0 txntype:-1 reqpath:n/a Error Path:/solr/collection= s/collection1/leaders/shard3 Error:KeeperErrorCode =3D NoNode for /solr/col= lections/collection1/leaders/shard3 [junit4:junit4] 2> 16421 T1979 oasc.ShardLeaderElectionContext.runLeaderP= rocess Running the leader process. [junit4:junit4] 2> 16421 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000a type:cr= eate cxid:0x46 zxid:0xa1 txntype:-1 reqpath:n/a Error Path:/solr/overseer E= rror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 16424 T1979 oasc.ShardLeaderElectionContext.waitForRep= licasToComeUp Enough replicas found to continue. [junit4:junit4] 2> 16425 T1979 oasc.ShardLeaderElectionContext.runLeaderP= rocess I may be the new leader - try and sync [junit4:junit4] 2> 16425 T1979 oasc.SyncStrategy.sync Sync replicas to ht= tp://127.0.0.1:35931/um_xr/collection1/ [junit4:junit4] 2> 16426 T1979 oasc.SyncStrategy.syncReplicas Sync Succes= s - now sync replicas to me [junit4:junit4] 2> 16426 T1979 oasc.SyncStrategy.syncToMe http://127.0.0.= 1:35931/um_xr/collection1/ has no replicas [junit4:junit4] 2> 16426 T1979 oasc.ShardLeaderElectionContext.runLeaderP= rocess I am the new leader: http://127.0.0.1:35931/um_xr/collection1/ [junit4:junit4] 2> 16427 T1979 oascc.SolrZkClient.makePath makePath: /col= lections/collection1/leaders/shard3 [junit4:junit4] 2> 16431 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000a type:cr= eate cxid:0x4f zxid:0xa4 txntype:-1 reqpath:n/a Error Path:/solr/overseer E= rror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 17101 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 17111 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 17111 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 17112 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 17112 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 17113 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 17135 T1979 oasc.ZkController.register We are http://1= 27.0.0.1:35931/um_xr/collection1/ and leader is http://127.0.0.1:35931/um_x= r/collection1/ [junit4:junit4] 2> 17135 T1979 oasc.ZkController.register No LogReplay ne= eded for core=3Dcollection1 baseURL=3Dhttp://127.0.0.1:35931/um_xr [junit4:junit4] 2> 17136 T1979 oasc.ZkController.checkRecovery I am the l= eader, no recovery necessary [junit4:junit4] 2> 17136 T1979 oasc.ZkController.publish publishing core= =3Dcollection1 state=3Dactive [junit4:junit4] 2> 17136 T1979 oasc.ZkController.publish numShards not fo= und on descriptor - reading it from system property [junit4:junit4] 2> 17139 T1979 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 17141 T1904 oass.SolrDispatchFilter.init user.dir=3D/m= nt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/= J1 [junit4:junit4] 2> 17141 T1904 oass.SolrDispatchFilter.init SolrDispatchF= ilter.init() done [junit4:junit4] 2> 17142 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnections=3D128&maxConnectionsPerHost=3D32&f= ollowRedirects=3Dfalse [junit4:junit4] 2> 17207 T1904 oejs.Server.doStart jetty-8.1.8.v20121106 [junit4:junit4] 2> 17210 T1904 oejs.AbstractConnector.doStart Started Sel= ectChannelConnector@127.0.0.1:36270 [junit4:junit4] 2> 17210 T1904 oass.SolrDispatchFilter.init SolrDispatchF= ilter.init() [junit4:junit4] 2> 17210 T1904 oasc.SolrResourceLoader.locateSolrHome JND= I not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 17211 T1904 oasc.SolrResourceLoader.locateSolrHome usi= ng system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-t= runk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkey= SafeLeaderTest-jetty4-1362466213213 [junit4:junit4] 2> 17211 T1904 oasc.CoreContainer$Initializer.initialize = looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-= Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeL= eaderTest-jetty4-1362466213213/solr.xml [junit4:junit4] 2> 17211 T1904 oasc.CoreContainer. New CoreContaine= r 477317376 [junit4:junit4] 2> 17212 T1904 oasc.CoreContainer.load Loading CoreContai= ner using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/so= lr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTes= t-jetty4-1362466213213/' [junit4:junit4] 2> 17212 T1904 oasc.SolrResourceLoader. new SolrRes= ourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-jetty4-1362466213213/' [junit4:junit4] 2> 17228 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting socketTimeout to: 120000 [junit4:junit4] 2> 17228 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting urlScheme to: http:// [junit4:junit4] 2> 17228 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting connTimeout to: 15000 [junit4:junit4] 2> 17229 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 17229 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting corePoolSize to: 0 [junit4:junit4] 2> 17229 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 17229 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maxThreadIdleTime to: 5 [junit4:junit4] 2> 17230 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting sizeOfQueue to: -1 [junit4:junit4] 2> 17230 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting fairnessPolicy to: false [junit4:junit4] 2> 17230 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000= &socketTimeout=3D120000&connTimeout=3D15000&retry=3Dfalse [junit4:junit4] 2> 17237 T1904 oasc.CoreContainer.load Registering Log Li= stener [junit4:junit4] 2> 17250 T1904 oasc.CoreContainer.initZooKeeper Zookeeper= client=3D127.0.0.1:33229/solr [junit4:junit4] 2> 17251 T1904 oasc.ZkController.checkChrootPath zkHost i= ncludes chroot [junit4:junit4] 2> 17251 T1904 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:33229 sessionTimeout=3D60000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@1cc12f31 [junit4:junit4] 2> 17252 T1991 oaz.ClientCnxn$SendThread.logStartConnect = Opening socket connection to server localhost.localdomain/127.0.0.1:33229. = Will not attempt to authenticate using SASL (access denied (javax.security.= auth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 17252 T1904 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 17253 T1991 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:33229, ini= tiating session [junit4:junit4] 2> 17253 T1906 oazs.NIOServerCnxnFactory.run Accepted soc= ket connection from /127.0.0.1:41363 [junit4:junit4] 2> 17253 T1906 oazs.ZooKeeperServer.processConnectRequest= Client attempting to establish new session at /127.0.0.1:41363 [junit4:junit4] 2> 17254 T1908 oazs.ZooKeeperServer.finishSessionInit Est= ablished session 0x13d39504a78000b with negotiated timeout 20000 for client= /127.0.0.1:41363 [junit4:junit4] 2> 17254 T1991 oaz.ClientCnxn$SendThread.onConnected Sess= ion establishment complete on server localhost.localdomain/127.0.0.1:33229,= sessionid =3D 0x13d39504a78000b, negotiated timeout =3D 20000 [junit4:junit4] 2> 17254 T1992 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1cc12f31 name:ZooKeeperConnect= ion Watcher:127.0.0.1:33229 got event WatchedEvent state:SyncConnected type= :None path:null path:null type:None [junit4:junit4] 2> 17255 T1904 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 17256 T1909 oazs.PrepRequestProcessor.pRequest2Txn Pro= cessed session termination for sessionid: 0x13d39504a78000b [junit4:junit4] 2> 17256 T1906 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:41363 which had sessionid 0x13d39504a7800= 0b [junit4:junit4] 2> 17257 T1992 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 17256 T1904 oaz.ZooKeeper.close Session: 0x13d39504a78= 000b closed [junit4:junit4] 2> 17257 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnections=3D500&maxConnectionsPerHost=3D16&s= ocketTimeout=3D120000&connTimeout=3D15000 [junit4:junit4] 2> 17260 T1904 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:33229/solr sessionTimeout=3D30000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1cc24984 [junit4:junit4] 2> 17261 T1993 oaz.ClientCnxn$SendThread.logStartConnect = Opening socket connection to server localhost.localdomain/127.0.0.1:33229. = Will not attempt to authenticate using SASL (access denied (javax.security.= auth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 17261 T1904 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 17261 T1906 oazs.NIOServerCnxnFactory.run Accepted soc= ket connection from /127.0.0.1:41364 [junit4:junit4] 2> 17261 T1993 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:33229, ini= tiating session [junit4:junit4] 2> 17262 T1906 oazs.ZooKeeperServer.processConnectRequest= Client attempting to establish new session at /127.0.0.1:41364 [junit4:junit4] 2> 17263 T1908 oazs.ZooKeeperServer.finishSessionInit Est= ablished session 0x13d39504a78000c with negotiated timeout 20000 for client= /127.0.0.1:41364 [junit4:junit4] 2> 17263 T1993 oaz.ClientCnxn$SendThread.onConnected Sess= ion establishment complete on server localhost.localdomain/127.0.0.1:33229,= sessionid =3D 0x13d39504a78000c, negotiated timeout =3D 20000 [junit4:junit4] 2> 17264 T1994 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1cc24984 name:ZooKeeperConnect= ion Watcher:127.0.0.1:33229/solr got event WatchedEvent state:SyncConnected= type:None path:null path:null type:None [junit4:junit4] 2> 17264 T1904 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 17265 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000c type:cr= eate cxid:0x1 zxid:0xb1 txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 17266 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000c type:cr= eate cxid:0x2 zxid:0xb2 txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 17268 T1904 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 18272 T1904 oasc.ZkController.createEphemeralLiveNode = Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36270_um_xr [junit4:junit4] 2> 18273 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000c type:de= lete cxid:0x9 zxid:0xb3 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/= 127.0.0.1:36270_um_xr Error:KeeperErrorCode =3D NoNode for /solr/live_nodes= /127.0.0.1:36270_um_xr [junit4:junit4] 2> 18275 T1904 oascc.SolrZkClient.makePath makePath: /liv= e_nodes/127.0.0.1:36270_um_xr [junit4:junit4] 2> 18279 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 18279 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 4) [junit4:junit4] 2> 18280 T1962 oascc.ZkStateReader$3.process Updating liv= e nodes... (5) [junit4:junit4] 2> 18280 T1933 oascc.ZkStateReader$3.process Updating liv= e nodes... (5) [junit4:junit4] 2> 18281 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 18281 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 18280 T1978 oascc.ZkStateReader$3.process Updating liv= e nodes... (5) [junit4:junit4] 2> 18282 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 18283 T1994 oascc.ZkStateReader$3.process Updating liv= e nodes... (5) [junit4:junit4] 2> 18283 T1926 oascc.ZkStateReader$3.process Updating liv= e nodes... (5) [junit4:junit4] 2> 18284 T1946 oascc.ZkStateReader$3.process Updating liv= e nodes... (5) [junit4:junit4] 2> 18293 T1995 oasc.CoreContainer.create Creating SolrCor= e 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-t= runk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkey= SafeLeaderTest-jetty4-1362466213213/collection1 [junit4:junit4] 2> 18293 T1995 oasc.ZkController.createCollectionZkNode C= heck for collection zkNode:collection1 [junit4:junit4] 2> 18294 T1995 oasc.ZkController.createCollectionZkNode C= ollection zkNode exists [junit4:junit4] 2> 18295 T1995 oasc.ZkController.readConfigName Load coll= ection config from:/collections/collection1 [junit4:junit4] 2> 18297 T1995 oasc.SolrResourceLoader. new SolrRes= ourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-jetty4-1362466213213/collection1/' [junit4:junit4] 2> 18298 T1995 oasc.SolrResourceLoader.replaceClassLoader= Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1= 362466213213/collection1/lib/README' to classloader [junit4:junit4] 2> 18299 T1995 oasc.SolrResourceLoader.replaceClassLoader= Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1= 362466213213/collection1/lib/classes/' to classloader [junit4:junit4] 2> 18342 T1995 oasc.SolrConfig. Using Lucene MatchV= ersion: LUCENE_50 [junit4:junit4] 2> 18382 T1995 oasc.SolrConfig. Loaded SolrConfig: = solrconfig.xml [junit4:junit4] 2> 18384 T1995 oass.IndexSchema.readSchema Reading Solr S= chema [junit4:junit4] 2> 18403 T1995 oass.IndexSchema.readSchema Schema name=3D= test [junit4:junit4] 2> 18619 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 18620 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":"shard3", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:35931_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:35931/um_xr"} [junit4:junit4] 2> 18624 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 18624 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 18624 T1994 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 18624 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 18624 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 18624 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 18848 T1995 oass.OpenExchangeRatesOrgProvider.init Ini= tialized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 18854 T1995 oass.IndexSchema.readSchema default search= field in schema is text [junit4:junit4] 2> 18858 T1995 oass.IndexSchema.readSchema unique key fie= ld: id [junit4:junit4] 2> 18873 T1995 oass.FileExchangeRateProvider.reload Reloa= ding exchange rates from file currency.xml [junit4:junit4] 2> 18876 T1995 oass.FileExchangeRateProvider.reload Reloa= ding exchange rates from file currency.xml [junit4:junit4] 2> 18881 T1995 oass.OpenExchangeRatesOrgProvider.reload R= eloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 18883 T1995 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 18883 T1995 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 18883 T1995 oass.OpenExchangeRatesOrgProvider.reload R= eloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 18885 T1995 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 18885 T1995 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 18885 T1995 oasc.SolrCore. [collection1] Opening= new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bu= ild/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jet= ty4-1362466213213/collection1/, dataDir=3D./org.apache.solr.cloud.ChaosMonk= eySafeLeaderTest-1362466196085/jetty4/ [junit4:junit4] 2> 18886 T1995 oasc.JmxMonitoredMap. JMX monitoring= is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxM= BeanServer@1c1fa491 [junit4:junit4] 2> 18886 T1995 oasc.SolrCore.initDirectoryFactory solr.St= andardDirectoryFactory [junit4:junit4] 2> 18887 T1995 oasc.CachingDirectoryFactory.get return ne= w directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bui= ld/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362= 466196085/jetty4 forceNew:false [junit4:junit4] 2> 18887 T1995 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty4 [junit4:junit4] 2> 18887 T1995 oasc.SolrCore.getNewIndexDir New index dir= ectory detected: old=3Dnull new=3D./org.apache.solr.cloud.ChaosMonkeySafeLe= aderTest-1362466196085/jetty4/index/ [junit4:junit4] 2> 18887 T1995 oasc.SolrCore.initIndex WARNING [collectio= n1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest= -1362466196085/jetty4/index' doesn't exist. Creating new index... [junit4:junit4] 2> 18888 T1995 oasc.CachingDirectoryFactory.get return ne= w directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bui= ld/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362= 466196085/jetty4/index forceNew:false [junit4:junit4] 2> 18893 T1995 oasc.SolrDeletionPolicy.onCommit SolrDelet= ionPolicy.onCommit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty4/index,segFN=3Dsegments_1,generation= =3D1,filenames=3D[segments_1] [junit4:junit4] 2> 18893 T1995 oasc.SolrDeletionPolicy.updateCommits newe= st commit =3D 1[segments_1] [junit4:junit4] 2> 18894 T1995 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty4/index [junit4:junit4] 2> 18896 T1995 oasup.UpdateRequestProcessorChain.init ins= erting DistributedUpdateProcessorFactory into updateRequestProcessorChain "= dedupe" [junit4:junit4] 2> 18897 T1995 oasup.UpdateRequestProcessorChain.init ins= erting DistributedUpdateProcessorFactory into updateRequestProcessorChain "= stored_sig" [junit4:junit4] 2> 18897 T1995 oasup.UpdateRequestProcessorChain.init ins= erting DistributedUpdateProcessorFactory into updateRequestProcessorChain "= distrib-dup-test-chain-implicit" [junit4:junit4] 2> 18898 T1995 oasc.RequestHandlers.initHandlersFromConfi= g created standard: solr.StandardRequestHandler [junit4:junit4] 2> 18899 T1995 oasc.RequestHandlers.initHandlersFromConfi= g created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 18900 T1995 oasc.RequestHandlers.initHandlersFromConfi= g adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 18900 T1995 oasc.RequestHandlers.initHandlersFromConfi= g created /replication: solr.ReplicationHandler [junit4:junit4] 2> 18901 T1995 oasc.RequestHandlers.initHandlersFromConfi= g created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 18902 T1995 oasc.RequestHandlers.initHandlersFromConfi= g created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 18906 T1995 oashl.XMLLoader.init xsltCacheLifetimeSeco= nds=3D60 [junit4:junit4] 2> 18911 T1995 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty4 [junit4:junit4] 2> 18913 T1995 oass.SolrIndexSearcher. Opening Sear= cher@1cd0fcc2 main [junit4:junit4] 2> 18913 T1995 oasu.CommitTracker. Hard AutoCommit:= disabled [junit4:junit4] 2> 18914 T1995 oasu.CommitTracker. Soft AutoCommit:= disabled [junit4:junit4] 2> 18923 T1996 oasc.SolrCore.registerSearcher [collection= 1] Registered new searcher Searcher@1cd0fcc2 main{StandardDirectoryReader(s= egments_1:1)} [junit4:junit4] 2> 18926 T1995 oasc.ZkController.publish publishing core= =3Dcollection1 state=3Ddown [junit4:junit4] 2> 18927 T1995 oasc.ZkController.publish numShards not fo= und on descriptor - reading it from system property [junit4:junit4] 2> 20130 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 20132 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":null, [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"down", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:36270_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:36270/um_xr"} [junit4:junit4] 2> 20132 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Collection already exists with numShards=3D3 [junit4:junit4] 2> 20132 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Assigning new node to shard shard=3Dshard1 [junit4:junit4] 2> 20138 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 20138 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 20138 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 20138 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 20138 T1994 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 20138 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 20929 T1995 oasc.CoreContainer.registerCore registerin= g core: collection1 [junit4:junit4] 2> 20929 T1995 oasc.ZkController.register Register replic= a - core:collection1 address:http://127.0.0.1:36270/um_xr collection:collec= tion1 shard:shard1 [junit4:junit4] 2> 20935 T1995 oasc.ZkController.register We are http://1= 27.0.0.1:36270/um_xr/collection1/ and leader is http://127.0.0.1:55383/um_x= r/collection1/ [junit4:junit4] 2> 20935 T1995 oasc.ZkController.register No LogReplay ne= eded for core=3Dcollection1 baseURL=3Dhttp://127.0.0.1:36270/um_xr [junit4:junit4] 2> 20936 T1995 oasc.ZkController.checkRecovery Core needs= to recover:collection1 [junit4:junit4] 2> 20936 T1995 oasu.DefaultSolrCoreState.doRecovery Runni= ng recovery - first canceling any ongoing recovery [junit4:junit4] 2> ASYNC NEW_CORE C438 name=3Dcollection1 org.apache.sol= r.core.SolrCore@1cd00aef url=3Dhttp://127.0.0.1:36270/um_xr/collection1 nod= e=3D127.0.0.1:36270_um_xr C438_STATE=3Dcoll:collection1 core:collection1 pr= ops:{state=3Ddown, core=3Dcollection1, collection=3Dcollection1, node_name= =3D127.0.0.1:36270_um_xr, base_url=3Dhttp://127.0.0.1:36270/um_xr} [junit4:junit4] 2> 20937 T1997 C438 P36270 oasc.RecoveryStrategy.run Star= ting recovery process. core=3Dcollection1 recoveringAfterStartup=3Dtrue [junit4:junit4] 2> 20937 T1997 C438 P36270 oasc.RecoveryStrategy.doRecove= ry ###### startupVersions=3D[] [junit4:junit4] 2> 20937 T1997 C438 P36270 oasc.ZkController.publish publ= ishing core=3Dcollection1 state=3Drecovering [junit4:junit4] 2> 20938 T1995 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 20938 T1997 C438 P36270 oasc.ZkController.publish numS= hards not found on descriptor - reading it from system property [junit4:junit4] 2> 20940 T1997 C438 P36270 oascsi.HttpClientUtil.createCl= ient Creating new http client, config:maxConnections=3D128&maxConnectionsPe= rHost=3D32&followRedirects=3Dfalse [junit4:junit4] 2> 20940 T1904 oass.SolrDispatchFilter.init user.dir=3D/m= nt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/= J1 [junit4:junit4] 2> 20940 T1904 oass.SolrDispatchFilter.init SolrDispatchF= ilter.init() done [junit4:junit4] 2> 20941 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnections=3D128&maxConnectionsPerHost=3D32&f= ollowRedirects=3Dfalse [junit4:junit4] 2> 21003 T1904 oejs.Server.doStart jetty-8.1.8.v20121106 [junit4:junit4] 2> 21005 T1904 oejs.AbstractConnector.doStart Started Sel= ectChannelConnector@127.0.0.1:44106 [junit4:junit4] 2> 21006 T1904 oass.SolrDispatchFilter.init SolrDispatchF= ilter.init() [junit4:junit4] 2> 21006 T1904 oasc.SolrResourceLoader.locateSolrHome JND= I not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 21006 T1904 oasc.SolrResourceLoader.locateSolrHome usi= ng system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-t= runk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkey= SafeLeaderTest-jetty5-1362466217011 [junit4:junit4] 2> 21007 T1904 oasc.CoreContainer$Initializer.initialize = looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-= Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeL= eaderTest-jetty5-1362466217011/solr.xml [junit4:junit4] 2> 21007 T1904 oasc.CoreContainer. New CoreContaine= r 483556922 [junit4:junit4] 2> 21007 T1904 oasc.CoreContainer.load Loading CoreContai= ner using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/so= lr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTes= t-jetty5-1362466217011/' [junit4:junit4] 2> 21008 T1904 oasc.SolrResourceLoader. new SolrRes= ourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-jetty5-1362466217011/' [junit4:junit4] 2> 21023 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting socketTimeout to: 120000 [junit4:junit4] 2> 21024 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting urlScheme to: http:// [junit4:junit4] 2> 21024 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting connTimeout to: 15000 [junit4:junit4] 2> 21024 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 21024 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting corePoolSize to: 0 [junit4:junit4] 2> 21025 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 21025 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maxThreadIdleTime to: 5 [junit4:junit4] 2> 21025 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting sizeOfQueue to: -1 [junit4:junit4] 2> 21025 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting fairnessPolicy to: false [junit4:junit4] 2> 21026 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000= &socketTimeout=3D120000&connTimeout=3D15000&retry=3Dfalse [junit4:junit4] 2> 21032 T1904 oasc.CoreContainer.load Registering Log Li= stener [junit4:junit4] 2> 21045 T1904 oasc.CoreContainer.initZooKeeper Zookeeper= client=3D127.0.0.1:33229/solr [junit4:junit4] 2> 21046 T1904 oasc.ZkController.checkChrootPath zkHost i= ncludes chroot [junit4:junit4] 2> 21046 T1904 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:33229 sessionTimeout=3D60000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@1cc28ab2 [junit4:junit4] 2> 21047 T1904 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 21047 T2008 oaz.ClientCnxn$SendThread.logStartConnect = Opening socket connection to server localhost.localdomain/127.0.0.1:33229. = Will not attempt to authenticate using SASL (access denied (javax.security.= auth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 21047 T2008 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:33229, ini= tiating session [junit4:junit4] 2> 21048 T1906 oazs.NIOServerCnxnFactory.run Accepted soc= ket connection from /127.0.0.1:41366 [junit4:junit4] 2> 21048 T1906 oazs.ZooKeeperServer.processConnectRequest= Client attempting to establish new session at /127.0.0.1:41366 [junit4:junit4] 2> 21049 T1908 oazs.ZooKeeperServer.finishSessionInit Est= ablished session 0x13d39504a78000d with negotiated timeout 20000 for client= /127.0.0.1:41366 [junit4:junit4] 2> 21049 T2008 oaz.ClientCnxn$SendThread.onConnected Sess= ion establishment complete on server localhost.localdomain/127.0.0.1:33229,= sessionid =3D 0x13d39504a78000d, negotiated timeout =3D 20000 [junit4:junit4] 2> 21049 T2009 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1cc28ab2 name:ZooKeeperConnect= ion Watcher:127.0.0.1:33229 got event WatchedEvent state:SyncConnected type= :None path:null path:null type:None [junit4:junit4] 2> 21049 T1904 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 21050 T1909 oazs.PrepRequestProcessor.pRequest2Txn Pro= cessed session termination for sessionid: 0x13d39504a78000d [junit4:junit4] 2> 21050 T2009 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 21050 T1904 oaz.ZooKeeper.close Session: 0x13d39504a78= 000d closed [junit4:junit4] 2> 21051 T1906 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:41366 which had sessionid 0x13d39504a7800= 0d [junit4:junit4] 2> 21051 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnections=3D500&maxConnectionsPerHost=3D16&s= ocketTimeout=3D120000&connTimeout=3D15000 [junit4:junit4] 2> 21054 T1904 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:33229/solr sessionTimeout=3D30000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1ce00525 [junit4:junit4] 2> 21055 T2010 oaz.ClientCnxn$SendThread.logStartConnect = Opening socket connection to server localhost.localdomain/127.0.0.1:33229. = Will not attempt to authenticate using SASL (access denied (javax.security.= auth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 21054 T1904 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 21055 T1906 oazs.NIOServerCnxnFactory.run Accepted soc= ket connection from /127.0.0.1:41367 [junit4:junit4] 2> 21055 T2010 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:33229, ini= tiating session [junit4:junit4] 2> 21056 T1906 oazs.ZooKeeperServer.processConnectRequest= Client attempting to establish new session at /127.0.0.1:41367 [junit4:junit4] 2> 21056 T1908 oazs.ZooKeeperServer.finishSessionInit Est= ablished session 0x13d39504a78000e with negotiated timeout 20000 for client= /127.0.0.1:41367 [junit4:junit4] 2> 21056 T2010 oaz.ClientCnxn$SendThread.onConnected Sess= ion establishment complete on server localhost.localdomain/127.0.0.1:33229,= sessionid =3D 0x13d39504a78000e, negotiated timeout =3D 20000 [junit4:junit4] 2> 21057 T2011 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1ce00525 name:ZooKeeperConnect= ion Watcher:127.0.0.1:33229/solr got event WatchedEvent state:SyncConnected= type:None path:null path:null type:None [junit4:junit4] 2> 21057 T1904 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 21058 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000e type:cr= eate cxid:0x1 zxid:0xc4 txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 21059 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000e type:cr= eate cxid:0x2 zxid:0xc5 txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 21061 T1904 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> 21645 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 21646 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":"shard1", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:36270_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:36270/um_xr"} [junit4:junit4] 2> 21666 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 21666 T1994 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 21666 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 21666 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 21666 T2011 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 21666 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 21666 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 22063 T1904 oasc.ZkController.createEphemeralLiveNode = Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44106_um_xr [junit4:junit4] 2> 22064 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a78000e type:de= lete cxid:0xb zxid:0xca txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/= 127.0.0.1:44106_um_xr Error:KeeperErrorCode =3D NoNode for /solr/live_nodes= /127.0.0.1:44106_um_xr [junit4:junit4] 2> 22065 T1904 oascc.SolrZkClient.makePath makePath: /liv= e_nodes/127.0.0.1:44106_um_xr [junit4:junit4] 2> 22067 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 22067 T2011 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 22067 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 5) [junit4:junit4] 2> 22068 T1933 oascc.ZkStateReader$3.process Updating liv= e nodes... (6) [junit4:junit4] 2> 22068 T1962 oascc.ZkStateReader$3.process Updating liv= e nodes... (6) [junit4:junit4] 2> 22068 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 22068 T1978 oascc.ZkStateReader$3.process Updating liv= e nodes... (6) [junit4:junit4] 2> 22068 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 22069 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 22068 T1994 oascc.ZkStateReader$3.process Updating liv= e nodes... (6) [junit4:junit4] 2> 22069 T1994 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 22069 T1946 oascc.ZkStateReader$3.process Updating liv= e nodes... (6) [junit4:junit4] 2> 22069 T2011 oascc.ZkStateReader$3.process Updating liv= e nodes... (6) [junit4:junit4] 2> 22070 T1926 oascc.ZkStateReader$3.process Updating liv= e nodes... (6) [junit4:junit4] 2> 22074 T2012 oasc.CoreContainer.create Creating SolrCor= e 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-t= runk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkey= SafeLeaderTest-jetty5-1362466217011/collection1 [junit4:junit4] 2> 22074 T2012 oasc.ZkController.createCollectionZkNode C= heck for collection zkNode:collection1 [junit4:junit4] 2> 22075 T2012 oasc.ZkController.createCollectionZkNode C= ollection zkNode exists [junit4:junit4] 2> 22075 T2012 oasc.ZkController.readConfigName Load coll= ection config from:/collections/collection1 [junit4:junit4] 2> 22076 T2012 oasc.SolrResourceLoader. new SolrRes= ourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-jetty5-1362466217011/collection1/' [junit4:junit4] 2> 22077 T2012 oasc.SolrResourceLoader.replaceClassLoader= Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1= 362466217011/collection1/lib/README' to classloader [junit4:junit4] 2> 22077 T2012 oasc.SolrResourceLoader.replaceClassLoader= Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1= 362466217011/collection1/lib/classes/' to classloader [junit4:junit4] 2> 22106 T2012 oasc.SolrConfig. Using Lucene MatchV= ersion: LUCENE_50 [junit4:junit4] 2> 22148 T2012 oasc.SolrConfig. Loaded SolrConfig: = solrconfig.xml [junit4:junit4] 2> 22149 T2012 oass.IndexSchema.readSchema Reading Solr S= chema [junit4:junit4] 2> 22154 T2012 oass.IndexSchema.readSchema Schema name=3D= test [junit4:junit4] 2> 22599 T2012 oass.OpenExchangeRatesOrgProvider.init Ini= tialized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 22607 T2012 oass.IndexSchema.readSchema default search= field in schema is text [junit4:junit4] 2> 22610 T2012 oass.IndexSchema.readSchema unique key fie= ld: id [junit4:junit4] 2> 22621 T2012 oass.FileExchangeRateProvider.reload Reloa= ding exchange rates from file currency.xml [junit4:junit4] 2> 22624 T2012 oass.FileExchangeRateProvider.reload Reloa= ding exchange rates from file currency.xml [junit4:junit4] 2> 22627 T2012 oass.OpenExchangeRatesOrgProvider.reload R= eloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 22628 T2012 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 22628 T2012 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 22628 T2012 oass.OpenExchangeRatesOrgProvider.reload R= eloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 22629 T2012 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 22629 T2012 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 22629 T2012 oasc.SolrCore. [collection1] Opening= new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bu= ild/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jet= ty5-1362466217011/collection1/, dataDir=3D./org.apache.solr.cloud.ChaosMonk= eySafeLeaderTest-1362466196085/jetty5/ [junit4:junit4] 2> 22630 T2012 oasc.JmxMonitoredMap. JMX monitoring= is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxM= BeanServer@1c1fa491 [junit4:junit4] 2> 22630 T2012 oasc.SolrCore.initDirectoryFactory solr.St= andardDirectoryFactory [junit4:junit4] 2> 22630 T2012 oasc.CachingDirectoryFactory.get return ne= w directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bui= ld/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362= 466196085/jetty5 forceNew:false [junit4:junit4] 2> 22631 T2012 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty5 [junit4:junit4] 2> 22631 T2012 oasc.SolrCore.getNewIndexDir New index dir= ectory detected: old=3Dnull new=3D./org.apache.solr.cloud.ChaosMonkeySafeLe= aderTest-1362466196085/jetty5/index/ [junit4:junit4] 2> 22631 T2012 oasc.SolrCore.initIndex WARNING [collectio= n1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest= -1362466196085/jetty5/index' doesn't exist. Creating new index... [junit4:junit4] 2> 22632 T2012 oasc.CachingDirectoryFactory.get return ne= w directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bui= ld/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362= 466196085/jetty5/index forceNew:false [junit4:junit4] 2> 22648 T2012 oasc.SolrDeletionPolicy.onCommit SolrDelet= ionPolicy.onCommit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty5/index,segFN=3Dsegments_1,generation= =3D1,filenames=3D[segments_1] [junit4:junit4] 2> 22648 T2012 oasc.SolrDeletionPolicy.updateCommits newe= st commit =3D 1[segments_1] [junit4:junit4] 2> 22649 T2012 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty5/index [junit4:junit4] 2> 22651 T2012 oasup.UpdateRequestProcessorChain.init ins= erting DistributedUpdateProcessorFactory into updateRequestProcessorChain "= dedupe" [junit4:junit4] 2> 22651 T2012 oasup.UpdateRequestProcessorChain.init ins= erting DistributedUpdateProcessorFactory into updateRequestProcessorChain "= stored_sig" [junit4:junit4] 2> 22652 T2012 oasup.UpdateRequestProcessorChain.init ins= erting DistributedUpdateProcessorFactory into updateRequestProcessorChain "= distrib-dup-test-chain-implicit" [junit4:junit4] 2> 22652 T2012 oasc.RequestHandlers.initHandlersFromConfi= g created standard: solr.StandardRequestHandler [junit4:junit4] 2> 22653 T2012 oasc.RequestHandlers.initHandlersFromConfi= g created /get: solr.RealTimeGetHandler [junit4:junit4] 2> 22653 T2012 oasc.RequestHandlers.initHandlersFromConfi= g adding lazy requestHandler: solr.ReplicationHandler [junit4:junit4] 2> 22653 T2012 oasc.RequestHandlers.initHandlersFromConfi= g created /replication: solr.ReplicationHandler [junit4:junit4] 2> 22654 T2012 oasc.RequestHandlers.initHandlersFromConfi= g created /update: solr.UpdateRequestHandler [junit4:junit4] 2> 22654 T2012 oasc.RequestHandlers.initHandlersFromConfi= g created /admin/: org.apache.solr.handler.admin.AdminHandlers [junit4:junit4] 2> 22656 T2012 oashl.XMLLoader.init xsltCacheLifetimeSeco= nds=3D60 [junit4:junit4] 2> 22659 T2012 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty5 [junit4:junit4] 2> 22659 T2012 oass.SolrIndexSearcher. Opening Sear= cher@1ce6c0ac main [junit4:junit4] 2> 22660 T2012 oasu.CommitTracker. Hard AutoCommit:= disabled [junit4:junit4] 2> 22660 T2012 oasu.CommitTracker. Soft AutoCommit:= disabled [junit4:junit4] 2> 22664 T2013 oasc.SolrCore.registerSearcher [collection= 1] Registered new searcher Searcher@1ce6c0ac main{StandardDirectoryReader(s= egments_1:1)} [junit4:junit4] 2> 22666 T2012 oasc.ZkController.publish publishing core= =3Dcollection1 state=3Ddown [junit4:junit4] 2> 22666 T2012 oasc.ZkController.publish numShards not fo= und on descriptor - reading it from system property [junit4:junit4] 2> 23174 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 23175 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":null, [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"down", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:44106_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:44106/um_xr"} [junit4:junit4] 2> 23175 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Collection already exists with numShards=3D3 [junit4:junit4] 2> 23175 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Assigning new node to shard shard=3Dshard2 [junit4:junit4] 2> 23266 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 23273 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 23273 T1994 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 23266 T2011 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 23273 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 23273 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 23266 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 23668 T2012 oasc.CoreContainer.registerCore registerin= g core: collection1 [junit4:junit4] 2> 23668 T2012 oasc.ZkController.register Register replic= a - core:collection1 address:http://127.0.0.1:44106/um_xr collection:collec= tion1 shard:shard2 [junit4:junit4] 2> 23671 T2012 oasc.ZkController.register We are http://1= 27.0.0.1:44106/um_xr/collection1/ and leader is http://127.0.0.1:35943/um_x= r/collection1/ [junit4:junit4] 2> 23671 T2012 oasc.ZkController.register No LogReplay ne= eded for core=3Dcollection1 baseURL=3Dhttp://127.0.0.1:44106/um_xr [junit4:junit4] 2> 23671 T2012 oasc.ZkController.checkRecovery Core needs= to recover:collection1 [junit4:junit4] 2> 23671 T2012 oasu.DefaultSolrCoreState.doRecovery Runni= ng recovery - first canceling any ongoing recovery [junit4:junit4] 2> ASYNC NEW_CORE C439 name=3Dcollection1 org.apache.sol= r.core.SolrCore@1cc5aafa url=3Dhttp://127.0.0.1:44106/um_xr/collection1 nod= e=3D127.0.0.1:44106_um_xr C439_STATE=3Dcoll:collection1 core:collection1 pr= ops:{state=3Ddown, core=3Dcollection1, collection=3Dcollection1, node_name= =3D127.0.0.1:44106_um_xr, base_url=3Dhttp://127.0.0.1:44106/um_xr} [junit4:junit4] 2> 23672 T2014 C439 P44106 oasc.RecoveryStrategy.run Star= ting recovery process. core=3Dcollection1 recoveringAfterStartup=3Dtrue [junit4:junit4] 2> 23672 T2014 C439 P44106 oasc.RecoveryStrategy.doRecove= ry ###### startupVersions=3D[] [junit4:junit4] 2> 23672 T2012 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 23672 T2014 C439 P44106 oasc.ZkController.publish publ= ishing core=3Dcollection1 state=3Drecovering [junit4:junit4] 2> 23672 T2014 C439 P44106 oasc.ZkController.publish numS= hards not found on descriptor - reading it from system property [junit4:junit4] 2> 23673 T1904 oass.SolrDispatchFilter.init user.dir=3D/m= nt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/= J1 [junit4:junit4] 2> 23673 T2014 C439 P44106 oascsi.HttpClientUtil.createCl= ient Creating new http client, config:maxConnections=3D128&maxConnectionsPe= rHost=3D32&followRedirects=3Dfalse [junit4:junit4] 2> 23673 T1904 oass.SolrDispatchFilter.init SolrDispatchF= ilter.init() done [junit4:junit4] 2> 23674 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnections=3D128&maxConnectionsPerHost=3D32&f= ollowRedirects=3Dfalse [junit4:junit4] 2> 23734 T1904 oejs.Server.doStart jetty-8.1.8.v20121106 [junit4:junit4] 2> 23736 T1904 oejs.AbstractConnector.doStart Started Sel= ectChannelConnector@127.0.0.1:40861 [junit4:junit4] 2> 23736 T1904 oass.SolrDispatchFilter.init SolrDispatchF= ilter.init() [junit4:junit4] 2> 23736 T1904 oasc.SolrResourceLoader.locateSolrHome JND= I not configured for solr (NoInitialContextEx) [junit4:junit4] 2> 23737 T1904 oasc.SolrResourceLoader.locateSolrHome usi= ng system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-t= runk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkey= SafeLeaderTest-jetty6-1362466219740 [junit4:junit4] 2> 23737 T1904 oasc.CoreContainer$Initializer.initialize = looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-= Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeL= eaderTest-jetty6-1362466219740/solr.xml [junit4:junit4] 2> 23737 T1904 oasc.CoreContainer. New CoreContaine= r 476964129 [junit4:junit4] 2> 23738 T1904 oasc.CoreContainer.load Loading CoreContai= ner using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/so= lr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTes= t-jetty6-1362466219740/' [junit4:junit4] 2> 23738 T1904 oasc.SolrResourceLoader. new SolrRes= ourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-jetty6-1362466219740/' [junit4:junit4] 2> 23755 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting socketTimeout to: 120000 [junit4:junit4] 2> 23755 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting urlScheme to: http:// [junit4:junit4] 2> 23755 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting connTimeout to: 15000 [junit4:junit4] 2> 23756 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maxConnectionsPerHost to: 20 [junit4:junit4] 2> 23756 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting corePoolSize to: 0 [junit4:junit4] 2> 23756 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maximumPoolSize to: 2147483647 [junit4:junit4] 2> 23756 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting maxThreadIdleTime to: 5 [junit4:junit4] 2> 23757 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting sizeOfQueue to: -1 [junit4:junit4] 2> 23757 T1904 oashc.HttpShardHandlerFactory.getParameter= Setting fairnessPolicy to: false [junit4:junit4] 2> 23757 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnectionsPerHost=3D20&maxConnections=3D10000= &socketTimeout=3D120000&connTimeout=3D15000&retry=3Dfalse [junit4:junit4] 2> 23764 T1904 oasc.CoreContainer.load Registering Log Li= stener [junit4:junit4] 2> 23778 T1904 oasc.CoreContainer.initZooKeeper Zookeeper= client=3D127.0.0.1:33229/solr [junit4:junit4] 2> 23778 T1904 oasc.ZkController.checkChrootPath zkHost i= ncludes chroot [junit4:junit4] 2> 23778 T1904 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:33229 sessionTimeout=3D60000 watcher=3Do= rg.apache.solr.common.cloud.ConnectionManager@1c84a568 [junit4:junit4] 2> 23779 T2025 oaz.ClientCnxn$SendThread.logStartConnect = Opening socket connection to server localhost.localdomain/127.0.0.1:33229. = Will not attempt to authenticate using SASL (access denied (javax.security.= auth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 23779 T1904 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 23780 T2025 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:33229, ini= tiating session [junit4:junit4] 2> 23780 T1906 oazs.NIOServerCnxnFactory.run Accepted soc= ket connection from /127.0.0.1:41369 [junit4:junit4] 2> 23780 T1906 oazs.ZooKeeperServer.processConnectRequest= Client attempting to establish new session at /127.0.0.1:41369 [junit4:junit4] 2> 23781 T1908 oazs.ZooKeeperServer.finishSessionInit Est= ablished session 0x13d39504a78000f with negotiated timeout 20000 for client= /127.0.0.1:41369 [junit4:junit4] 2> 23781 T2025 oaz.ClientCnxn$SendThread.onConnected Sess= ion establishment complete on server localhost.localdomain/127.0.0.1:33229,= sessionid =3D 0x13d39504a78000f, negotiated timeout =3D 20000 [junit4:junit4] 2> 23781 T2026 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1c84a568 name:ZooKeeperConnect= ion Watcher:127.0.0.1:33229 got event WatchedEvent state:SyncConnected type= :None path:null path:null type:None [junit4:junit4] 2> 23782 T1904 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 23782 T1909 oazs.PrepRequestProcessor.pRequest2Txn Pro= cessed session termination for sessionid: 0x13d39504a78000f [junit4:junit4] 2> 23783 T1906 oazs.NIOServerCnxn.closeSock Closed socket= connection for client /127.0.0.1:41369 which had sessionid 0x13d39504a7800= 0f [junit4:junit4] 2> 23783 T2026 oaz.ClientCnxn$EventThread.run EventThread= shut down [junit4:junit4] 2> 23783 T1904 oaz.ZooKeeper.close Session: 0x13d39504a78= 000f closed [junit4:junit4] 2> 23784 T1904 oascsi.HttpClientUtil.createClient Creatin= g new http client, config:maxConnections=3D500&maxConnectionsPerHost=3D16&s= ocketTimeout=3D120000&connTimeout=3D15000 [junit4:junit4] 2> 23786 T1904 oaz.ZooKeeper. Initiating client con= nection, connectString=3D127.0.0.1:33229/solr sessionTimeout=3D30000 watche= r=3Dorg.apache.solr.common.cloud.ConnectionManager@1c84411b [junit4:junit4] 2> 23787 T2027 oaz.ClientCnxn$SendThread.logStartConnect = Opening socket connection to server localhost.localdomain/127.0.0.1:33229. = Will not attempt to authenticate using SASL (access denied (javax.security.= auth.AuthPermission getLoginConfiguration)) [junit4:junit4] 2> 23787 T1904 oascc.ConnectionManager.waitForConnected W= aiting for client to connect to ZooKeeper [junit4:junit4] 2> 23788 T2027 oaz.ClientCnxn$SendThread.primeConnection = Socket connection established to localhost.localdomain/127.0.0.1:33229, ini= tiating session [junit4:junit4] 2> 23788 T1906 oazs.NIOServerCnxnFactory.run Accepted soc= ket connection from /127.0.0.1:41370 [junit4:junit4] 2> 23788 T1906 oazs.ZooKeeperServer.processConnectRequest= Client attempting to establish new session at /127.0.0.1:41370 [junit4:junit4] 2> 23789 T1908 oazs.ZooKeeperServer.finishSessionInit Est= ablished session 0x13d39504a780010 with negotiated timeout 20000 for client= /127.0.0.1:41370 [junit4:junit4] 2> 23789 T2027 oaz.ClientCnxn$SendThread.onConnected Sess= ion establishment complete on server localhost.localdomain/127.0.0.1:33229,= sessionid =3D 0x13d39504a780010, negotiated timeout =3D 20000 [junit4:junit4] 2> 23789 T2028 oascc.ConnectionManager.process Watcher or= g.apache.solr.common.cloud.ConnectionManager@1c84411b name:ZooKeeperConnect= ion Watcher:127.0.0.1:33229/solr got event WatchedEvent state:SyncConnected= type:None path:null path:null type:None [junit4:junit4] 2> 23789 T1904 oascc.ConnectionManager.waitForConnected C= lient is connected to ZooKeeper [junit4:junit4] 2> 23790 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a780010 type:cr= eate cxid:0x1 zxid:0xd7 txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 23791 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a780010 type:cr= eate cxid:0x2 zxid:0xd8 txntype:-1 reqpath:n/a Error Path:/solr/overseer Er= ror:KeeperErrorCode =3D NodeExists for /solr/overseer [junit4:junit4] 2> 23792 T1904 oascc.ZkStateReader.createClusterStateWatc= hersAndUpdate Updating cluster state from ZooKeeper...=20 [junit4:junit4] 2> C438_STATE=3Dcoll:collection1 core:collection1 props:= {shard=3Dshard1, state=3Drecovering, core=3Dcollection1, collection=3Dcolle= ction1, node_name=3D127.0.0.1:36270_um_xr, base_url=3Dhttp://127.0.0.1:3627= 0/um_xr} [junit4:junit4] 2> 23948 T1997 C438 P36270 oasc.RecoveryStrategy.doRecove= ry Attempting to PeerSync from http://127.0.0.1:55383/um_xr/collection1/ co= re=3Dcollection1 - recoveringAfterStartup=3Dtrue [junit4:junit4] 2> 23948 T1997 C438 P36270 oasu.PeerSync.sync PeerSync: c= ore=3Dcollection1 url=3Dhttp://127.0.0.1:36270/um_xr START replicas=3D[http= ://127.0.0.1:55383/um_xr/collection1/] nUpdates=3D100 [junit4:junit4] 2> 23949 T1997 C438 P36270 oasu.PeerSync.sync WARNING no = frame of reference to tell of we've missed updates [junit4:junit4] 2> 23949 T1997 C438 P36270 oasc.RecoveryStrategy.doRecove= ry PeerSync Recovery was not successful - trying replication. core=3Dcollec= tion1 [junit4:junit4] 2> 23950 T1997 C438 P36270 oasc.RecoveryStrategy.doRecove= ry Starting Replication Recovery. core=3Dcollection1 [junit4:junit4] 2> 23950 T1997 C438 P36270 oasc.RecoveryStrategy.doRecove= ry Begin buffering updates. core=3Dcollection1 [junit4:junit4] 2> 23950 T1997 C438 P36270 oasu.UpdateLog.bufferUpdates S= tarting to buffer updates. FSUpdateLog{state=3DACTIVE, tlog=3Dnull} [junit4:junit4] 2> 23951 T1997 C438 P36270 oasc.RecoveryStrategy.replicat= e Attempting to replicate from http://127.0.0.1:55383/um_xr/collection1/. c= ore=3Dcollection1 [junit4:junit4] 2> 23951 T1997 C438 P36270 oascsi.HttpClientUtil.createCl= ient Creating new http client, config:maxConnections=3D128&maxConnectionsPe= rHost=3D32&followRedirects=3Dfalse [junit4:junit4] 2> ASYNC NEW_CORE C440 name=3Dcollection1 org.apache.sol= r.core.SolrCore@1c78e3ae url=3Dhttp://127.0.0.1:55383/um_xr/collection1 nod= e=3D127.0.0.1:55383_um_xr C440_STATE=3Dcoll:collection1 core:collection1 pr= ops:{shard=3Dshard1, state=3Dactive, core=3Dcollection1, collection=3Dcolle= ction1, node_name=3D127.0.0.1:55383_um_xr, base_url=3Dhttp://127.0.0.1:5538= 3/um_xr, leader=3Dtrue} [junit4:junit4] 2> 23954 T1940 C440 P55383 REQ /get {getVersions=3D100&di= strib=3Dfalse&wt=3Djavabin&qt=3D/get&version=3D2} status=3D0 QTime=3D0=20 [junit4:junit4] 2> 23958 T1941 C440 P55383 oasu.DirectUpdateHandler2.comm= it start commit{,optimize=3Dfalse,openSearcher=3Dfalse,waitSearcher=3Dtrue,= expungeDeletes=3Dfalse,softCommit=3Dfalse,prepareCommit=3Dfalse} [junit4:junit4] 2> 23958 T1941 C440 P55383 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty1 [junit4:junit4] 2> 23960 T1941 C440 P55383 oasc.SolrDeletionPolicy.onInit= SolrDeletionPolicy.onInit: commits:num=3D1 [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty1/index,segFN=3Dsegments_1,generation= =3D1,filenames=3D[segments_1] [junit4:junit4] 2> 23960 T1941 C440 P55383 oasc.SolrDeletionPolicy.update= Commits newest commit =3D 1[segments_1] [junit4:junit4] 2> 23966 T1941 C440 P55383 oasc.SolrDeletionPolicy.onComm= it SolrDeletionPolicy.onCommit: commits:num=3D2 [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty1/index,segFN=3Dsegments_1,generation= =3D1,filenames=3D[segments_1] [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty1/index,segFN=3Dsegments_2,generation= =3D2,filenames=3D[segments_2] [junit4:junit4] 2> 23966 T1941 C440 P55383 oasc.SolrDeletionPolicy.update= Commits newest commit =3D 2[segments_2] [junit4:junit4] 2> 23967 T1941 C440 P55383 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty1 [junit4:junit4] 2> 23967 T1941 C440 P55383 oass.SolrIndexSearcher. = Opening Searcher@1c930a9a realtime [junit4:junit4] 2> 23967 T1941 C440 P55383 oasu.DirectUpdateHandler2.comm= it end_commit_flush [junit4:junit4] 2> 23968 T1941 C440 P55383 /update {waitSearcher=3Dtrue&o= penSearcher=3Dfalse&commit=3Dtrue&wt=3Djavabin&commit_end_point=3Dtrue&vers= ion=3D2&softCommit=3Dfalse} {commit=3D} 0 10 [junit4:junit4] 2> 23968 T1997 C438 P36270 oash.ReplicationHandler.inform= Commits will be reserved for 10000 [junit4:junit4] 2> 23968 T1997 C438 P36270 oash.SnapPuller. No val= ue set for 'pollInterval'. Timer Task not started. [junit4:junit4] 2> 23970 T1942 C440 P55383 oash.ReplicationHandler.inform= Commits will be reserved for 10000 [junit4:junit4] 2> 23970 T1942 C440 P55383 REQ /replication {command=3Din= dexversion&qt=3D/replication&wt=3Djavabin&version=3D2} status=3D0 QTime=3D1= =20 [junit4:junit4] 2> 23970 T1997 C438 P36270 oash.SnapPuller.fetchLatestInd= ex Master's generation: 2 [junit4:junit4] 2> 23970 T1997 C438 P36270 oash.SnapPuller.fetchLatestInd= ex Slave's generation: 1 [junit4:junit4] 2> 23971 T1997 C438 P36270 oash.SnapPuller.fetchLatestInd= ex Starting replication process [junit4:junit4] 2> 23972 T1940 C440 P55383 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty1 [junit4:junit4] 2> 23973 T1940 C440 P55383 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty1/index [junit4:junit4] 2> 23973 T1940 C440 P55383 REQ /replication {command=3Dfi= lelist&qt=3D/replication&wt=3Djavabin&generation=3D2&version=3D2} status=3D= 0 QTime=3D1=20 [junit4:junit4] 2> 23974 T1997 C438 P36270 oash.SnapPuller.fetchLatestInd= ex Number of files in latest index in master: 1 [junit4:junit4] 2> 23974 T1997 C438 P36270 oasc.CachingDirectoryFactory.g= et return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-1362466196085/jetty4/index.20130305075020038 forceNew:false [junit4:junit4] 2> 23975 T1997 C438 P36270 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty4 [junit4:junit4] 2> 23975 T1997 C438 P36270 oash.SnapPuller.fetchLatestInd= ex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenk= ins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apac= he.solr.cloud.ChaosMonkeySafeLeaderTest-1362466196085/jetty4/index.20130305= 075020038 lockFactory=3Dorg.apache.lucene.store.NativeFSLockFactory@1c93b29= 5 fullCopy=3Dfalse [junit4:junit4] 2> 23977 T1941 C440 P55383 REQ /replication {file=3Dsegme= nts_2&command=3Dfilecontent&checksum=3Dtrue&wt=3Dfilestream&qt=3D/replicati= on&generation=3D2} status=3D0 QTime=3D0=20 [junit4:junit4] 2> 23978 T1997 C438 P36270 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty4/index [junit4:junit4] 2> 23978 T1997 C438 P36270 oash.SnapPuller.fetchLatestInd= ex Total time taken for download : 0 secs [junit4:junit4] 2> 23986 T1997 C438 P36270 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty4 [junit4:junit4] 2> 23987 T1997 C438 P36270 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty4 [junit4:junit4] 2> 23987 T1997 C438 P36270 oasu.DefaultSolrCoreState.newI= ndexWriter Creating new IndexWriter... [junit4:junit4] 2> 23987 T1997 C438 P36270 oasu.DefaultSolrCoreState.newI= ndexWriter Waiting until IndexWriter is unused... core=3Dcollection1 [junit4:junit4] 2> 23988 T1997 C438 P36270 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty4 [junit4:junit4] 2> 23989 T1997 C438 P36270 oasc.SolrDeletionPolicy.onInit= SolrDeletionPolicy.onInit: commits:num=3D2 [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty4/index,segFN=3Dsegments_1,generation= =3D1,filenames=3D[segments_1] [junit4:junit4] 2> =09=09commit{dir=3D/mnt/ssd/jenkins/workspace/Lucene-S= olr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMon= keySafeLeaderTest-1362466196085/jetty4/index,segFN=3Dsegments_2,generation= =3D2,filenames=3D[segments_2] [junit4:junit4] 2> 23989 T1997 C438 P36270 oasc.SolrDeletionPolicy.update= Commits newest commit =3D 2[segments_2] [junit4:junit4] 2> 23990 T1997 C438 P36270 oasu.DefaultSolrCoreState.newI= ndexWriter New IndexWriter is ready to be used. [junit4:junit4] 2> 23990 T1997 C438 P36270 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty4 [junit4:junit4] 2> 23990 T1997 C438 P36270 oass.SolrIndexSearcher. = Opening Searcher@1c945650 main [junit4:junit4] 2> 23991 T1996 oasc.SolrCore.registerSearcher [collection= 1] Registered new searcher Searcher@1c945650 main{StandardDirectoryReader(s= egments_2:1:nrt)} [junit4:junit4] 2> 23991 T1996 oasc.CachingDirectoryFactory.close Releasi= ng directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/= solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1362466= 196085/jetty4/index [junit4:junit4] 2> 23992 T1997 C438 P36270 oash.SnapPuller.fetchLatestInd= ex removing temporary index download directory files org.apache.lucene.stor= e.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/bui= ld/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-136246= 6196085/jetty4/index.20130305075020038 lockFactory=3Dorg.apache.lucene.stor= e.NativeFSLockFactory@1c93b295 [junit4:junit4] 2> 23992 T1997 C438 P36270 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty4/index.20130305075020038 [junit4:junit4] 2> 23993 T1997 C438 P36270 oasc.CachingDirectoryFactory.c= lose Releasing directory:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux= /solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeader= Test-1362466196085/jetty4/index [junit4:junit4] 2> 23993 T1997 C438 P36270 oasc.RecoveryStrategy.replay N= o replay needed. core=3Dcollection1 [junit4:junit4] 2> 23993 T1997 C438 P36270 oasc.RecoveryStrategy.doRecove= ry Replication Recovery was successful - registering as Active. core=3Dcoll= ection1 [junit4:junit4] 2> 23993 T1997 C438 P36270 oasc.ZkController.publish publ= ishing core=3Dcollection1 state=3Dactive [junit4:junit4] 2> 23993 T1997 C438 P36270 oasc.ZkController.publish numS= hards not found on descriptor - reading it from system property [junit4:junit4] 2> 23995 T1997 C438 P36270 oasc.RecoveryStrategy.doRecove= ry Finished recovery process. core=3Dcollection1 [junit4:junit4] 2> 24780 T1927 oascc.ZkStateReader.updateClusterState Upd= ating cloud state from ZooKeeper...=20 [junit4:junit4] 2> 24781 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":"shard2", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"recovering", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:44106_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:44106/um_xr"} [junit4:junit4] 2> 24787 T1927 oasc.Overseer$ClusterStateUpdater.updateSt= ate Update state numShards=3D3 message=3D{ [junit4:junit4] 2> =09 "operation":"state", [junit4:junit4] 2> =09 "core_node_name":null, [junit4:junit4] 2> =09 "numShards":"3", [junit4:junit4] 2> =09 "shard":"shard1", [junit4:junit4] 2> =09 "roles":null, [junit4:junit4] 2> =09 "state":"active", [junit4:junit4] 2> =09 "core":"collection1", [junit4:junit4] 2> =09 "collection":"collection1", [junit4:junit4] 2> =09 "node_name":"127.0.0.1:36270_um_xr", [junit4:junit4] 2> =09 "base_url":"http://127.0.0.1:36270/um_xr"} [junit4:junit4] 2> 24806 T2028 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24806 T2011 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24806 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24807 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24806 T1994 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24806 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24806 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24806 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clu= sterstate.json, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24807 T1904 oasc.ZkController.createEphemeralLiveNode = Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40861_um_xr [junit4:junit4] 2> 24810 T1909 oazs.PrepRequestProcessor.pRequest Got use= r-level KeeperException when processing sessionid:0x13d39504a780010 type:de= lete cxid:0xb zxid:0xe0 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/= 127.0.0.1:40861_um_xr Error:KeeperErrorCode =3D NoNode for /solr/live_nodes= /127.0.0.1:40861_um_xr [junit4:junit4] 2> 24811 T1904 oascc.SolrZkClient.makePath makePath: /liv= e_nodes/127.0.0.1:40861_um_xr [junit4:junit4] 2> 24815 T2028 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24815 T2011 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24815 T1946 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24816 T1994 oascc.ZkStateReader$3.process Updating liv= e nodes... (7) [junit4:junit4] 2> 24816 T1962 oascc.ZkStateReader$3.process Updating liv= e nodes... (7) [junit4:junit4] 2> 24816 T1926 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 6) [junit4:junit4] 2> 24816 T1978 oascc.ZkStateReader$3.process Updating liv= e nodes... (7) [junit4:junit4] 2> 24817 T1962 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 7) [junit4:junit4] 2> 24817 T1978 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 7) [junit4:junit4] 2> 24818 T1933 oascc.ZkStateReader$3.process Updating liv= e nodes... (7) [junit4:junit4] 2> 24816 T1994 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 7) [junit4:junit4] 2> 24818 T1933 oascc.ZkStateReader$2.process A cluster st= ate change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:= /live_nodes, has occurred - updating... (live nodes size: 7) [junit4:junit4] 2> 24820 T1946 oascc.ZkStateReader$3.process Updating liv= e nodes... (7) [junit4:junit4] 2> 24820 T2028 oascc.ZkStateReader$3.process Updating liv= e nodes... (7) [junit4:junit4] 2> 24820 T2011 oascc.ZkStateReader$3.process Updating liv= e nodes... (7) [junit4:junit4] 2> 24821 T1926 oascc.ZkStateReader$3.process Updating liv= e nodes... (7) [junit4:junit4] 2> 24830 T2031 oasc.CoreContainer.create Creating SolrCor= e 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-t= runk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkey= SafeLeaderTest-jetty6-1362466219740/collection1 [junit4:junit4] 2> 24831 T2031 oasc.ZkController.createCollectionZkNode C= heck for collection zkNode:collection1 [junit4:junit4] 2> 24832 T2031 oasc.ZkController.createCollectionZkNode C= ollection zkNode exists [junit4:junit4] 2> 24832 T2031 oasc.ZkController.readConfigName Load coll= ection config from:/collections/collection1 [junit4:junit4] 2> 24834 T2031 oasc.SolrResourceLoader. new SolrRes= ourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Li= nux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLea= derTest-jetty6-1362466219740/collection1/' [junit4:junit4] 2> 24835 T2031 oasc.SolrResourceLoader.replaceClassLoader= Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1= 362466219740/collection1/lib/README' to classloader [junit4:junit4] 2> 24836 T2031 oasc.SolrResourceLoader.replaceClassLoader= Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build= /solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1= 362466219740/collection1/lib/classes/' to classloader [junit4:junit4] 2> 24883 T2031 oasc.SolrConfig. Using Lucene MatchV= ersion: LUCENE_50 [junit4:junit4] 2> 24923 T2031 oasc.SolrConfig. Loaded SolrConfig: = solrconfig.xml [junit4:junit4] 2> 24924 T2031 oass.IndexSchema.readSchema Reading Solr S= chema [junit4:junit4] 2> 24929 T2031 oass.IndexSchema.readSchema Schema name=3D= test [junit4:junit4] 2> 25389 T2031 oass.OpenExchangeRatesOrgProvider.init Ini= tialized with rates=3Dopen-exchange-rates.json, refreshInterval=3D1440. [junit4:junit4] 2> 25395 T2031 oass.IndexSchema.readSchema default search= field in schema is text [junit4:junit4] 2> 25398 T2031 oass.IndexSchema.readSchema unique key fie= ld: id [junit4:junit4] 2> 25411 T2031 oass.FileExchangeRateProvider.reload Reloa= ding exchange rates from file currency.xml [junit4:junit4] 2> 25415 T2031 oass.FileExchangeRateProvider.reload Reloa= ding exchange rates from file currency.xml [junit4:junit4] 2> 25419 T2031 oass.OpenExchangeRatesOrgProvider.reload R= eloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 25420 T2031 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 25420 T2031 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 25420 T2031 oass.OpenExchangeRatesOrgProvider.reload R= eloading exchange rates from open-exchange-rates.json [junit4:junit4] 2> 25421 T2031 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Unknown key IMPORTANT NOTE [junit4:junit4] 2> 25421 T2031 oass.OpenExchangeRatesOrgProvider$OpenExch= angeRates. WARNING Expected key, got STRING [junit4:junit4] 2> 25422 T2031 oasc.SolrCore. [collection1] Opening= new SolrCore at /mnt/ssd/jenkins/work [...truncated too long message...] it4] 2> =09 at org.apache.solr.cloud.ZkController.register(ZkContr= oller.java:766) [junit4:junit4] 2> =09 at org.apache.solr.cloud.ZkController$1.com= mand(ZkController.java:212) [junit4:junit4] 2> =09 at org.apache.solr.common.cloud.ConnectionM= anager$1.update(ConnectionManager.java:117) [junit4:junit4] 2> =09 at org.apache.solr.common.cloud.DefaultConn= ectionStrategy.reconnect(DefaultConnectionStrategy.java:46) [junit4:junit4] 2> =09 at org.apache.solr.common.cloud.ConnectionM= anager.process(ConnectionManager.java:91) [junit4:junit4] 2> =09 at org.apache.zookeeper.ClientCnxn$EventThr= ead.processEvent(ClientCnxn.java:519) [junit4:junit4] 2> =09 at org.apache.zookeeper.ClientCnxn$EventThr= ead.run(ClientCnxn.java:495) [junit4:junit4] 2> 139957 T1903 ccr.ThreadLeakControl.tryToInterruptAll S= tarting to interrupt leaked threads: [junit4:junit4] 2> =09 1) Thread[id=3D2123, name=3DTEST-ChaosMonkeySafe= LeaderTest.testDistribSearch-seed#[E22BFD0C90735963]-EventThread, state=3DT= IMED_WAITING, group=3DTGRP-ChaosMonkeySafeLeaderTest] [junit4:junit4] 2> 139957 T2123 oasc.ZkController.getLeader SEVERE Error = getting leader from zk java.lang.InterruptedException: sleep interrupted [junit4:junit4] 2> =09at java.lang.Thread.sleep(Native Method) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkStateReader.getLe= aderRetry(ZkStateReader.java:428) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ZkStateReader.getLe= aderUrl(ZkStateReader.java:404) [junit4:junit4] 2> =09at org.apache.solr.cloud.ZkController.getLeader(ZkC= ontroller.java:833) [junit4:junit4] 2> =09at org.apache.solr.cloud.ZkController.register(ZkCo= ntroller.java:766) [junit4:junit4] 2> =09at org.apache.solr.cloud.ZkController$1.command(ZkC= ontroller.java:212) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ConnectionManager$1= .update(ConnectionManager.java:117) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.DefaultConnectionSt= rategy.reconnect(DefaultConnectionStrategy.java:46) [junit4:junit4] 2> =09at org.apache.solr.common.cloud.ConnectionManager.p= rocess(ConnectionManager.java:91) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.proc= essEvent(ClientCnxn.java:519) [junit4:junit4] 2> =09at org.apache.zookeeper.ClientCnxn$EventThread.run(= ClientCnxn.java:495) [junit4:junit4] 2>=20 [junit4:junit4] 2> 139958 T2123 oasc.SolrException.log SEVERE Error regis= tering SolrCore:org.apache.solr.common.SolrException: Error getting leader = from zk [junit4:junit4] 2> =09=09at org.apache.solr.cloud.ZkController.getLeader(= ZkController.java:854) [junit4:junit4] 2> =09=09at org.apache.solr.cloud.ZkController.register(Z= kController.java:766) [junit4:junit4] 2> =09=09at org.apache.solr.cloud.ZkController$1.command(= ZkController.java:212) [junit4:junit4] 2> =09=09at org.apache.solr.common.cloud.ConnectionManage= r$1.update(ConnectionManager.java:117) [junit4:junit4] 2> =09=09at org.apache.solr.common.cloud.DefaultConnectio= nStrategy.reconnect(DefaultConnectionStrategy.java:48) [junit4:junit4] 2> =09=09at org.apache.solr.common.cloud.ConnectionManage= r.process(ConnectionManager.java:91) [junit4:junit4] 2> =09=09at org.apache.zookeeper.ClientCnxn$EventThread.p= rocessEvent(ClientCnxn.java:519) [junit4:junit4] 2> =09=09at org.apache.zookeeper.ClientCnxn$EventThread.r= un(ClientCnxn.java:495) [junit4:junit4] 2> =09Caused by: java.lang.InterruptedException: sleep in= terrupted [junit4:junit4] 2> =09=09at java.lang.Thread.sleep(Native Method) [junit4:junit4] 2> =09=09at org.apache.solr.common.cloud.ZkStateReader.ge= tLeaderRetry(ZkStateReader.java:428) [junit4:junit4] 2> =09=09at org.apache.solr.common.cloud.ZkStateReader.ge= tLeaderUrl(ZkStateReader.java:404) [junit4:junit4] 2> =09=09at org.apache.solr.cloud.ZkController.getLeader(= ZkController.java:833) [junit4:junit4] 2> =09=09at org.apache.solr.cloud.ZkController.register(Z= kController.java:766) [junit4:junit4] 2> =09=09at org.apache.solr.cloud.ZkController$1.command(= ZkController.java:212) [junit4:junit4] 2> =09=09at org.apache.solr.common.cloud.ConnectionManage= r$1.update(ConnectionManager.java:117) [junit4:junit4] 2> =09=09at org.apache.solr.common.cloud.DefaultConnectio= nStrategy.reconnect(DefaultConnectionStrategy.java:46) [junit4:junit4] 2> =09=09... 3 more [junit4:junit4] 2> =09 [junit4:junit4] 2> 139958 T2123 oascc.DefaultConnectionStrategy.reconnect= Reconnected to ZooKeeper [junit4:junit4] 2> 139959 T2123 oascc.ConnectionManager.process Connected= :true [junit4:junit4] 2> 139959 T2123 oaz.ClientCnxn$EventThread.run EventThrea= d shut down [junit4:junit4] 2> 139959 T1903 ccr.ThreadLeakControl.tryToInterruptAll A= ll leaked threads terminated. [junit4:junit4] 2> NOTE: test params are: codec=3DLucene42: {range_facet_= l=3DPostingsFormat(name=3DMockSep), text=3DPostingsFormat(name=3DMemory doP= ackFST=3D false), _version_=3DPostingsFormat(name=3DMockSep), rnd_b=3DPosti= ngsFormat(name=3DAsserting), intDefault=3DPostingsFormat(name=3DMockSep), i= d=3DPostingsFormat(name=3DAsserting), timestamp=3DPostingsFormat(name=3DMoc= kSep), a_t=3DPostingsFormat(name=3DMockSep), range_facet_sl=3DPostingsForma= t(name=3DMemory doPackFST=3D false), range_facet_si=3DMockFixedIntBlock(blo= ckSize=3D477), other_tl1=3DPostingsFormat(name=3DMockSep), multiDefault=3DM= ockFixedIntBlock(blockSize=3D477), a_si=3DMockFixedIntBlock(blockSize=3D477= )}, docValues:{timestamp=3DDocValuesFormat(name=3DAsserting)}, sim=3DDefaul= tSimilarity, locale=3Dda_DK, timezone=3DEurope/Zurich [junit4:junit4] 2> NOTE: Linux 3.2.0-38-generic amd64/Oracle Corporation = 1.6.0_33 (64-bit)/cpus=3D8,threads=3D2,free=3D186482088,total=3D346308608 [junit4:junit4] 2> NOTE: All tests run in this JVM: [ClusterStateTest, Te= stArbitraryIndexDir, TestMergePolicyConfig, TestLMJelinekMercerSimilarityFa= ctory, BadIndexSchemaTest, OpenCloseCoreStressTest, OutputWriterTest, Colle= ctionsAPIDistributedZkTest, JSONWriterTest, LeaderElectionIntegrationTest, = TestFastOutputStream, EchoParamsTest, TestCollationField, SolrPluginUtilsTe= st, SpellPossibilityIteratorTest, SolrIndexConfigTest, DirectUpdateHandlerT= est, TestAnalyzedSuggestions, SortByFunctionTest, TestReload, OpenExchangeR= atesOrgProviderTest, TestDistributedSearch, NumericFieldsTest, TestZkChroot= , TestHashPartitioner, HighlighterTest, NotRequiredUniqueKeyTest, XsltUpdat= eRequestHandlerTest, DocValuesMultiTest, TestQueryTypes, HardAutoCommitTest= , TestAtomicUpdateErrorCases, TestFoldingMultitermQuery, ConvertedLegacyTes= t, UniqFieldsUpdateProcessorFactoryTest, CurrencyFieldXmlFileTest, TestProp= Inject, IndexBasedSpellCheckerTest, CurrencyFieldOpenExchangeTest, TestOmit= Positions, TestPartialUpdateDeduplication, DebugComponentTest, TermVectorCo= mponentDistributedTest, RequestHandlersTest, RequiredFieldsTest, TestStress= Reorder, TestCSVLoader, TestUtils, DistanceFunctionTest, TestBinaryField, U= pdateParamsTest, ChaosMonkeySafeLeaderTest] [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=3DChaosMonk= eySafeLeaderTest -Dtests.seed=3DE22BFD0C90735963 -Dtests.multiplier=3D3 -Dt= ests.slow=3Dtrue -Dtests.locale=3Dda_DK -Dtests.timezone=3DEurope/Zurich -D= tests.file.encoding=3DUS-ASCII [junit4:junit4] ERROR 0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<< [junit4:junit4] > Throwable #1: com.carrotsearch.randomizedtesting.Threa= dLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.Chaos= MonkeySafeLeaderTest:=20 [junit4:junit4] > 1) Thread[id=3D2123, name=3DTEST-ChaosMonkeySafeLea= derTest.testDistribSearch-seed#[E22BFD0C90735963]-EventThread, state=3DTIME= D_WAITING, group=3DTGRP-ChaosMonkeySafeLeaderTest] [junit4:junit4] > at java.lang.Thread.sleep(Native Method) [junit4:junit4] > at org.apache.solr.common.cloud.ZkStateReader.= getLeaderRetry(ZkStateReader.java:428) [junit4:junit4] > at org.apache.solr.common.cloud.ZkStateReader.= getLeaderUrl(ZkStateReader.java:404) [junit4:junit4] > at org.apache.solr.cloud.ZkController.getLeade= r(ZkController.java:833) [junit4:junit4] > at org.apache.solr.cloud.ZkController.register= (ZkController.java:766) [junit4:junit4] > at org.apache.solr.cloud.ZkController$1.comman= d(ZkController.java:212) [junit4:junit4] > at org.apache.solr.common.cloud.ConnectionMana= ger$1.update(ConnectionManager.java:117) [junit4:junit4] > at org.apache.solr.common.cloud.DefaultConnect= ionStrategy.reconnect(DefaultConnectionStrategy.java:46) [junit4:junit4] > at org.apache.solr.common.cloud.ConnectionMana= ger.process(ConnectionManager.java:91) [junit4:junit4] > at org.apache.zookeeper.ClientCnxn$EventThread= .processEvent(ClientCnxn.java:519) [junit4:junit4] > at org.apache.zookeeper.ClientCnxn$EventThread= .run(ClientCnxn.java:495) [junit4:junit4] > =09at __randomizedtesting.SeedInfo.seed([E22BFD0C90735= 963]:0) [junit4:junit4] Completed on J1 in 140.03s, 1 test, 1 error <<< FAILURES! [...truncated 466 lines...] BUILD FAILED /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:381: The follo= wing error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:361: The follo= wing error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The follow= ing error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:183: The = following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:44= 9: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:= 1213: The following error occurred while executing this line: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:= 877: There were test failures: 261 suites, 1105 tests, 1 suite-level error,= 13 ignored (7 assumptions) Total time: 91 minutes 33 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording test results Description set: Java: 64bit/jrockit-jdk1.6.0_33-R28.2.4-4.1.0 -XnoOpt Email was triggered for: Failure Sending email for trigger: Failure ------=_Part_16_1303718807.1362467103398 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org ------=_Part_16_1303718807.1362467103398--