lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_31) - Build # 11865 - Failure!
Date Tue, 24 Feb 2015 06:53:11 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11865/
Java: 32bit/jdk1.8.0_31 -server -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
There were too many update fails - we expect it can happen, but shouldn't easily

Stack Trace:
java.lang.AssertionError: There were too many update fails - we expect it can happen, but shouldn't easily
	at __randomizedtesting.SeedInfo.seed([4EE07024D3E2062D:C6B44FFE7D1E6BD5]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:222)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:940)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:915)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9624 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/init-core-data-001
   [junit4]   2> 499210 T4337 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 499215 T4337 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 499216 T4338 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 499316 T4337 oasc.ZkTestServer.run start zk server on port:40017
   [junit4]   2> 499317 T4337 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 499317 T4337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 499319 T4345 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e7741d name:ZooKeeperConnection Watcher:127.0.0.1:40017 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 499320 T4337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 499320 T4337 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 499321 T4337 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 499323 T4337 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 499324 T4337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 499325 T4348 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f2be94 name:ZooKeeperConnection Watcher:127.0.0.1:40017/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 499326 T4337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 499326 T4337 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 499327 T4337 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 499329 T4337 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 499331 T4337 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 499332 T4337 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 499334 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 499334 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 499337 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 499338 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 499340 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 499340 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 499342 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 499343 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 499344 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 499345 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 499346 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 499347 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 499348 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 499349 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 499350 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 499351 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 499353 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 499353 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 499355 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 499355 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 499357 T4337 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 499357 T4337 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 499702 T4337 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001/cores/collection1
   [junit4]   2> 499703 T4337 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 499704 T4337 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@130e8d8{/,null,AVAILABLE}
   [junit4]   2> 499706 T4337 oejs.AbstractConnector.doStart Started ServerConnector@e99541{HTTP/1.1}{127.0.0.1:41064}
   [junit4]   2> 499707 T4337 oejs.Server.doStart Started @500688ms
   [junit4]   2> 499707 T4337 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 499708 T4337 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 499708 T4337 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001
   [junit4]   2> 499708 T4337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001/'
   [junit4]   2> 499718 T4337 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001/solr.xml
   [junit4]   2> 499722 T4337 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001/cores
   [junit4]   2> 499722 T4337 oasc.CoreContainer.<init> New CoreContainer 18433421
   [junit4]   2> 499723 T4337 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001/]
   [junit4]   2> 499723 T4337 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 499723 T4337 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 499724 T4337 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 499724 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 499724 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 499724 T4337 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 499725 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 499725 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 499725 T4337 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 499725 T4337 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 499726 T4337 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 499726 T4337 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 499726 T4337 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 499727 T4337 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 499727 T4337 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 499727 T4337 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40017/solr
   [junit4]   2> 499727 T4337 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 499728 T4337 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 499728 T4337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 499729 T4362 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@183df59 name:ZooKeeperConnection Watcher:127.0.0.1:40017 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 499730 T4337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 499730 T4337 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 499732 T4337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 499732 T4365 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b8b810 name:ZooKeeperConnection Watcher:127.0.0.1:40017/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 499733 T4337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 499734 T4337 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 499735 T4337 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 499736 T4337 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 499738 T4337 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 499739 T4337 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 499741 T4337 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 499741 T4337 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41064_
   [junit4]   2> 499742 T4337 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41064_
   [junit4]   2> 499743 T4337 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 499744 T4337 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 499745 T4337 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 499746 T4337 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:41064_
   [junit4]   2> 499746 T4337 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 499747 T4337 oasc.Overseer.start Overseer (id=93373034452942851-127.0.0.1:41064_-n_0000000000) starting
   [junit4]   2> 499749 T4337 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 499753 T4337 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 499753 T4366 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 499754 T4366 oasc.Overseer$ClusterStateUpdater.run ERROR Exception in Overseer work queue loop org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /clusterstate.json
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:343)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:340)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.constructState(ZkStateReader.java:448)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:527)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:259)
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:164)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 499758 T4367 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 499759 T4337 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 499761 T4337 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 499762 T4337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 499763 T4337 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001/cores
   [junit4]   2> 499764 T4337 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001/cores/collection1/
   [junit4]   2> 499764 T4337 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 499765 T4369 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 499765 T4369 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 499765 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 499766 T4369 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 499766 T4366 oasc.Overseer$ClusterStateUpdater.run ERROR Exception in Overseer main queue loop java.lang.NullPointerException
   [junit4]   2> 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:221)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 499768 T4366 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41064",
   [junit4]   2> 	  "node_name":"127.0.0.1:41064_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 499769 T4366 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41064",
   [junit4]   2> 	  "node_name":"127.0.0.1:41064_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 499769 T4366 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 499769 T4366 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 499770 T4365 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 499771 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 500766 T4369 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 500766 T4369 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 500767 T4369 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 500767 T4369 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 500768 T4369 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 500768 T4369 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001/cores/collection1/'
   [junit4]   2> 500775 T4369 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 500777 T4369 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 500780 T4369 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 500784 T4369 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 500799 T4369 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 500800 T4369 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 500803 T4369 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 500873 T4369 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 500874 T4369 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 500875 T4369 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 500877 T4369 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 500886 T4369 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 500886 T4369 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 500887 T4369 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/control-001/cores/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/control/data/
   [junit4]   2> 500887 T4369 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1be4b9a
   [junit4]   2> 500888 T4369 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/control/data
   [junit4]   2> 500888 T4369 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/control/data/index/
   [junit4]   2> 500888 T4369 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 500888 T4369 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/control/data/index
   [junit4]   2> 500889 T4369 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=18.021484375, floorSegmentMB=0.6484375, forceMergeDeletesPctAllowed=15.31012635848657, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 501037 T4369 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 501038 T4369 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 501039 T4369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 501039 T4369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 501039 T4369 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 501040 T4369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 501040 T4369 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 501040 T4369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 501040 T4369 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 501040 T4369 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 501041 T4369 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 501041 T4369 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 501041 T4369 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 501041 T4369 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 501042 T4369 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 501042 T4369 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 501042 T4369 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 501042 T4369 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 501043 T4369 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 501043 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 501043 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 501043 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 501044 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 501044 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 501044 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 501044 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 501045 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 501045 T4369 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 501045 T4369 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 501046 T4369 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 501047 T4369 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 501048 T4369 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 501048 T4369 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 501049 T4369 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 501049 T4369 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 501050 T4369 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 501050 T4369 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4976682559212793]
   [junit4]   2> 501051 T4369 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 501051 T4369 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 501051 T4369 oass.SolrIndexSearcher.<init> Opening Searcher@faa585[collection1] main
   [junit4]   2> 501051 T4369 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 501052 T4369 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 501052 T4369 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 501053 T4369 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 501053 T4369 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 501053 T4369 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 501053 T4369 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 501054 T4369 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 501054 T4369 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 501054 T4369 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 501054 T4369 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 501054 T4370 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@faa585[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 501055 T4373 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41064 collection:control_collection shard:shard1
   [junit4]   2> 501055 T4337 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 501056 T4337 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 501056 T4373 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 501056 T4337 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 501057 T4337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 501058 T4376 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1829982 name:ZooKeeperConnection Watcher:127.0.0.1:40017/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 501058 T4337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 501059 T4337 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 501060 T4337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 501060 T4373 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 501062 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 501062 T4373 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 501062 T4373 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1602 name=collection1 org.apache.solr.core.SolrCore@1fb3855 url=http://127.0.0.1:41064/collection1 node=127.0.0.1:41064_ C1602_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:41064, node_name=127.0.0.1:41064_, state=down}
   [junit4]   2> 501062 T4373 C1602 P41064 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41064/collection1/
   [junit4]   2> 501063 T4373 C1602 P41064 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 501063 T4373 C1602 P41064 oasc.SyncStrategy.syncToMe http://127.0.0.1:41064/collection1/ has no replicas
   [junit4]   2> 501063 T4373 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41064/collection1/ shard1
   [junit4]   2> 501063 T4373 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 501063 T4337 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 501063 T4366 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 501065 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 501066 T4366 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41064",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 501168 T4365 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 501168 T4376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 501216 T4373 oasc.ZkController.register We are http://127.0.0.1:41064/collection1/ and leader is http://127.0.0.1:41064/collection1/
   [junit4]   2> 501217 T4373 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41064
   [junit4]   2> 501217 T4373 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 501217 T4373 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 501217 T4373 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 501218 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 501219 T4366 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41064",
   [junit4]   2> 	  "node_name":"127.0.0.1:41064_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 501219 T4366 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:41064",
   [junit4]   2> 	  "node_name":"127.0.0.1:41064_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 501313 T4337 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001/cores/collection1
   [junit4]   2> 501314 T4337 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 501315 T4337 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 501316 T4337 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@b492cf{/,null,AVAILABLE}
   [junit4]   2> 501317 T4337 oejs.AbstractConnector.doStart Started ServerConnector@1ab0473{HTTP/1.1}{127.0.0.1:45353}
   [junit4]   2> 501317 T4337 oejs.Server.doStart Started @502298ms
   [junit4]   2> 501318 T4337 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 501318 T4337 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 501319 T4337 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001
   [junit4]   2> 501319 T4337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001/'
   [junit4]   2> 501321 T4365 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 501322 T4376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 501330 T4337 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001/solr.xml
   [junit4]   2> 501334 T4337 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001/cores
   [junit4]   2> 501334 T4337 oasc.CoreContainer.<init> New CoreContainer 14696549
   [junit4]   2> 501335 T4337 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001/]
   [junit4]   2> 501335 T4337 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 501335 T4337 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 501335 T4337 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 501336 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 501336 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 501336 T4337 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 501336 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 501337 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 501337 T4337 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 501337 T4337 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 501337 T4337 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 501338 T4337 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 501338 T4337 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 501338 T4337 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 501339 T4337 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 501339 T4337 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40017/solr
   [junit4]   2> 501339 T4337 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 501339 T4337 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 501340 T4337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 501341 T4390 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@130f6c9 name:ZooKeeperConnection Watcher:127.0.0.1:40017 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 501341 T4337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 501342 T4337 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 501343 T4337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 501344 T4393 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1086512 name:ZooKeeperConnection Watcher:127.0.0.1:40017/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 501344 T4337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 501348 T4337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 502350 T4337 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45353_
   [junit4]   2> 502350 T4337 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45353_
   [junit4]   2> 502353 T4337 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 502354 T4337 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001/cores
   [junit4]   2> 502355 T4337 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001/cores/collection1/
   [junit4]   2> 502355 T4337 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 502356 T4394 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 502356 T4394 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 502357 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 502357 T4394 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 502357 T4366 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45353",
   [junit4]   2> 	  "node_name":"127.0.0.1:45353_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 502358 T4366 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45353",
   [junit4]   2> 	  "node_name":"127.0.0.1:45353_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 502358 T4366 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 502358 T4366 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 502460 T4365 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 502460 T4393 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 502460 T4376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 503357 T4394 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 503357 T4394 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 503358 T4394 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 503358 T4394 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 503359 T4394 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 503359 T4394 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001/cores/collection1/'
   [junit4]   2> 503373 T4394 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 503375 T4394 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 503378 T4394 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 503381 T4394 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 503389 T4394 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 503390 T4394 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 503393 T4394 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 503463 T4394 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 503465 T4394 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 503465 T4394 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 503467 T4394 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 503476 T4394 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 503476 T4394 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 503477 T4394 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-1-001/cores/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty1/
   [junit4]   2> 503477 T4394 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1be4b9a
   [junit4]   2> 503477 T4394 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty1
   [junit4]   2> 503478 T4394 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty1/index/
   [junit4]   2> 503478 T4394 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 503478 T4394 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty1/index
   [junit4]   2> 503478 T4394 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=18.021484375, floorSegmentMB=0.6484375, forceMergeDeletesPctAllowed=15.31012635848657, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 503492 T4394 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 503492 T4394 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 503494 T4394 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 503494 T4394 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 503494 T4394 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 503494 T4394 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 503494 T4394 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 503495 T4394 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 503495 T4394 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 503495 T4394 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 503495 T4394 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 503495 T4394 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 503496 T4394 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 503496 T4394 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 503496 T4394 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 503497 T4394 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 503497 T4394 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 503498 T4394 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 503498 T4394 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 503498 T4394 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 503499 T4394 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 503499 T4394 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 503499 T4394 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 503500 T4394 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 503500 T4394 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 503500 T4394 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 503501 T4394 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 503501 T4394 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 503501 T4394 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 503502 T4394 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 503503 T4394 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 503504 T4394 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 503504 T4394 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 503505 T4394 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 503505 T4394 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 503505 T4394 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 503506 T4394 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4976682559212793]
   [junit4]   2> 503507 T4394 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 503507 T4394 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 503507 T4394 oass.SolrIndexSearcher.<init> Opening Searcher@1a620d[collection1] main
   [junit4]   2> 503508 T4394 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 503508 T4394 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 503509 T4394 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 503509 T4394 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 503509 T4394 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 503509 T4394 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 503510 T4394 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 503510 T4394 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 503510 T4394 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 503510 T4394 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 503511 T4395 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a620d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 503511 T4394 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 503512 T4398 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45353 collection:collection1 shard:shard1
   [junit4]   2> 503512 T4337 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 503512 T4337 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 503512 T4398 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 503515 T4398 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 503516 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 503516 T4398 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 503516 T4366 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 503516 T4398 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1603 name=collection1 org.apache.solr.core.SolrCore@1e69aeb url=http://127.0.0.1:45353/collection1 node=127.0.0.1:45353_ C1603_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:45353, node_name=127.0.0.1:45353_, state=down}
   [junit4]   2> 503517 T4398 C1603 P45353 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45353/collection1/
   [junit4]   2> 503517 T4398 C1603 P45353 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 503517 T4398 C1603 P45353 oasc.SyncStrategy.syncToMe http://127.0.0.1:45353/collection1/ has no replicas
   [junit4]   2> 503517 T4398 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45353/collection1/ shard1
   [junit4]   2> 503517 T4398 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 503519 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 503520 T4366 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45353",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 503621 T4365 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 503621 T4393 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 503621 T4376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 503670 T4398 oasc.ZkController.register We are http://127.0.0.1:45353/collection1/ and leader is http://127.0.0.1:45353/collection1/
   [junit4]   2> 503670 T4398 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45353
   [junit4]   2> 503670 T4398 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 503670 T4398 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 503670 T4398 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 503671 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 503672 T4366 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45353",
   [junit4]   2> 	  "node_name":"127.0.0.1:45353_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 503673 T4366 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45353",
   [junit4]   2> 	  "node_name":"127.0.0.1:45353_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 503775 T4365 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 503775 T4393 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 503775 T4376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 503801 T4337 oas.SolrTestCaseJ4.writeCoreProperties Writing core.properties file to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001/cores/collection1
   [junit4]   2> 503802 T4337 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 503803 T4337 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 503811 T4337 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@13e34c7{/,null,AVAILABLE}
   [junit4]   2> 503812 T4337 oejs.AbstractConnector.doStart Started ServerConnector@f82f5b{HTTP/1.1}{127.0.0.1:42660}
   [junit4]   2> 503812 T4337 oejs.Server.doStart Started @504793ms
   [junit4]   2> 503812 T4337 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 503813 T4337 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 503813 T4337 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001
   [junit4]   2> 503813 T4337 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001/'
   [junit4]   2> 503823 T4337 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001/solr.xml
   [junit4]   2> 503827 T4337 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001/cores
   [junit4]   2> 503827 T4337 oasc.CoreContainer.<init> New CoreContainer 10518480
   [junit4]   2> 503827 T4337 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001/]
   [junit4]   2> 503828 T4337 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 503828 T4337 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 503828 T4337 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 503829 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 503829 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 503829 T4337 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 503829 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 503829 T4337 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 503830 T4337 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 503830 T4337 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 503830 T4337 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 503831 T4337 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 503831 T4337 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 503831 T4337 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 503831 T4337 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 503832 T4337 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40017/solr
   [junit4]   2> 503832 T4337 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 503832 T4337 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 503833 T4337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 503834 T4412 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1463439 name:ZooKeeperConnection Watcher:127.0.0.1:40017 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 503834 T4337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 503834 T4337 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 503836 T4337 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 503836 T4415 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cade69 name:ZooKeeperConnection Watcher:127.0.0.1:40017/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 503837 T4337 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 503840 T4337 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 504842 T4337 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42660_
   [junit4]   2> 504843 T4337 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42660_
   [junit4]   2> 504845 T4337 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 504847 T4337 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001/cores
   [junit4]   2> 504848 T4337 oasc.CorePropertiesLocator.discoverUnder Found core collection1 in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001/cores/collection1/
   [junit4]   2> 504848 T4337 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 504849 T4416 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 504849 T4416 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 504849 T4416 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 504849 T4365 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 504850 T4366 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42660",
   [junit4]   2> 	  "node_name":"127.0.0.1:42660_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 6
   [junit4]   2> 504851 T4366 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42660",
   [junit4]   2> 	  "node_name":"127.0.0.1:42660_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 504851 T4366 oasco.ReplicaMutator.updateState Collection already exists with numShards=1
   [junit4]   2> 504851 T4366 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 504953 T4365 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 504953 T4415 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 504953 T4376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 504953 T4393 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 505850 T4416 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 505850 T4416 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 505851 T4416 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 505851 T4416 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 505852 T4416 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 505852 T4416 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001/cores/collection1/'
   [junit4]   2> 505859 T4416 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 505860 T4416 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 505864 T4416 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 505867 T4416 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 505874 T4416 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 505875 T4416 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 505879 T4416 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 505948 T4416 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 505949 T4416 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 505950 T4416 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 505951 T4416 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 505960 T4416 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 505960 T4416 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 505961 T4416 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/shard-2-001/cores/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty2/
   [junit4]   2> 505961 T4416 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1be4b9a
   [junit4]   2> 505961 T4416 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty2
   [junit4]   2> 505962 T4416 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty2/index/
   [junit4]   2> 505962 T4416 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 505962 T4416 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty2/index
   [junit4]   2> 505963 T4416 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=34, maxMergedSegmentMB=18.021484375, floorSegmentMB=0.6484375, forceMergeDeletesPctAllowed=15.31012635848657, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 505974 T4416 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 505974 T4416 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 505976 T4416 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 505976 T4416 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 505976 T4416 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 505977 T4416 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 505977 T4416 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 505977 T4416 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 505977 T4416 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 505977 T4416 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 505977 T4416 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 505978 T4416 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 505978 T4416 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 505978 T4416 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 505979 T4416 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 505979 T4416 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 505979 T4416 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 505980 T4416 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 505980 T4416 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 505981 T4416 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 505981 T4416 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 505981 T4416 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 505982 T4416 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 505982 T4416 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 505982 T4416 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 505983 T4416 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 505983 T4416 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 505983 T4416 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 505983 T4416 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 505985 T4416 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 505985 T4416 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 505986 T4416 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 505987 T4416 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 505987 T4416 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 505988 T4416 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 505992 T4416 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 505994 T4416 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=43, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4976682559212793]
   [junit4]   2> 505994 T4416 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 505994 T4416 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 505994 T4416 oass.SolrIndexSearcher.<init> Opening Searcher@13ead3e[collection1] main
   [junit4]   2> 505995 T4416 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 505996 T4416 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 505996 T4416 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 505996 T4416 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 505996 T4416 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 505997 T4416 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 505997 T4416 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 505997 T4416 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 505997 T4416 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 505998 T4416 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 505998 T4417 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13ead3e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 505998 T4416 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 505999 T4420 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42660 collection:collection1 shard:shard1
   [junit4]   2> 505999 T4337 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 506000 T4337 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 506001 T4420 oasc.ZkController.register We are http://127.0.0.1:42660/collection1/ and leader is http://127.0.0.1:45353/collection1/
   [junit4]   2> 506001 T4420 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42660
   [junit4]   2> 506001 T4420 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 506001 T4420 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> ASYNC  NEW_CORE C1604 name=collection1 org.apache.solr.core.SolrCore@c5cd22 url=http://127.0.0.1:42660/collection1 node=127.0.0.1:42660_ C1604_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:42660, node_name=127.0.0.1:42660_, state=down}
   [junit4]   2> 506002 T4421 C1604 P42660 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 506002 T4421 C1604 P42660 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 506002 T4421 C1604 P42660 oasc.RecoveryStrategy.doRecovery Publishing state of core collection1 as recovering, leader is http://127.0.0.1:45353/collection1/ and I am http://127.0.0.1:42660/collection1/
   [junit4]   2> 506003 T4421 C1604 P42660 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
   [junit4]   2> 506003 T4421 C1604 P42660 oasc.ZkCont

[...truncated too long message...]

ve_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=28452,transaction_logs_total_number=1}
   [junit4]   2> 529677 T4473 oasco.ReplicaMutator.updateState Update state numShards=null message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node3",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52278",
   [junit4]   2> 	  "node_name":"127.0.0.1:52278_",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 529677 T4337 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 529677 T4337 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 529678 T4337 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 529678 T4440 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 529679 T4337 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 529694 T4337 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 529694 T4337 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty3/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty3/index;done=false>>]
   [junit4]   2> 529695 T4337 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty3/index
   [junit4]   2> 529695 T4337 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty3 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty3;done=false>>]
   [junit4]   2> 529696 T4337 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001/tempDir-001/jetty3
   [junit4]   2> 529696 T4337 oasc.Overseer.close Overseer (id=93373034452942858-127.0.0.1:52278_-n_0000000003) closing
   [junit4]   2> 529697 T4473 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:52278_
   [junit4]   2> 531199 T4440 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 531236 T4337 oejs.AbstractConnector.doStop Stopped ServerConnector@c5825f{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 531237 T4337 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@1d7733c{/,null,UNAVAILABLE}
   [junit4]   2> 531238 T4337 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40017 40017
   [junit4]   2> 532011 T4338 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40017 40017
   [junit4]   2> 532013 T4338 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		5	/solr/aliases.json
   [junit4]   2> 		5	/solr/clusterstate.json
   [junit4]   2> 		4	/solr/configs/conf1
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent data watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		2	/solr/overseer_elect/election/93373034452942851-127.0.0.1:41064_-n_0000000000
   [junit4]   2> 		2	/solr/collections/collection1/leader_elect/shard1/election/93373034452942854-core_node1-n_0000000000
   [junit4]   2> 		2	/solr/overseer_elect/election/93373034452942854-127.0.0.1:45353_-n_0000000001
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		5	/solr/live_nodes
   [junit4]   2> 		2	/solr/overseer/queue
   [junit4]   2> 		2	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=test -Dtests.seed=4EE07024D3E2062D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de_AT -Dtests.timezone=Africa/Accra -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 32.8s J0 | ChaosMonkeyNothingIsSafeTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There were too many update fails - we expect it can happen, but shouldn't easily
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4EE07024D3E2062D:C6B44FFE7D1E6BD5]:0)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:222)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:940)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:915)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 532022 T4337 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeyNothingIsSafeTest 4EE07024D3E2062D-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {rnd_b=PostingsFormat(name=LuceneVarGapDocFreqInterval), _version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_i=PostingsFormat(name=LuceneVarGapDocFreqInterval), id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=de_AT, timezone=Africa/Accra
   [junit4]   2> NOTE: Linux 3.13.0-45-generic i386/Oracle Corporation 1.8.0_31 (32-bit)/cpus=8,threads=1,free=141041144,total=295174144
   [junit4]   2> NOTE: All tests run in this JVM: [SolrCoreCheckLockOnStartupTest, TestBulkSchemaConcurrent, MultiTermTest, TestObjectReleaseTracker, TestStressLucene, TestXIncludeConfig, ScriptEngineTest, DocValuesMultiTest, OverseerTest, SystemInfoHandlerTest, ClusterStateUpdateTest, SharedFSAutoReplicaFailoverTest, TestSearcherReuse, SuggestComponentTest, TestInitParams, TestSolrIndexConfig, TestDistribDocBasedVersion, TestPhraseSuggestions, TestLFUCache, TestOrdValues, TestPostingsSolrHighlighter, TestFieldCollectionResource, TestCSVLoader, TestExpandComponent, IgnoreCommitOptimizeUpdateProcessorFactoryTest, SliceStateTest, TestInitQParser, BadComponentTest, TestSolrXml, TestCursorMarkWithoutUniqueKey, TestCopyFieldCollectionResource, TestStressRecovery, TestDocBasedVersionConstraints, TestPartialUpdateDeduplication, QueryElevationComponentTest, TestFaceting, DistributedFacetPivotLongTailTest, SimpleFacetsTest, HardAutoCommitTest, TestBlobHandler, SuggesterTest, ShardRoutingTest, RequestHandlersTest, AlternateDirectoryTest, LeaderInitiatedRecoveryOnCommitTest, TestOmitPositions, CloudMLTQParserTest, TestStressUserVersions, TestSolrDynamicMBean, TestFunctionQuery, SampleTest, DirectSolrConnectionTest, TestBinaryField, TestLRUCache, OverseerStatusTest, CoreAdminRequestStatusTest, DistributedSuggestComponentTest, PreAnalyzedFieldTest, TestWriterPerf, TestShardHandlerFactory, PrimUtilsTest, TestCollationField, PathHierarchyTokenizerFactoryTest, TestMiniSolrCloudClusterSSL, DistributedFacetPivotLargeTest, TestSolr4Spatial2, AddSchemaFieldsUpdateProcessorFactoryTest, TestFieldTypeResource, JSONWriterTest, TestSchemaVersionResource, DistanceFunctionTest, TestManagedResource, CursorPagingTest, SpatialHeatmapFacetsTest, CollectionsAPIAsyncDistributedZkTest, TestCloudSchemaless, TestConfigReload, XmlUpdateRequestHandlerTest, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed on J0 in 32.85s, 1 test, 1 failure <<< FAILURES!

[...truncated 894 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:191: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:509: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1349: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:957: There were test failures: 466 suites, 1860 tests, 1 failure, 46 ignored (21 assumptions)

Total time: 60 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_31 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Mime
View raw message