lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-Tests-4.x-Java7 - Build # 1962 - Failure
Date Sun, 13 Apr 2014 22:36:37 GMT
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java7/1962/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
document count mismatch.  control=82 sum(shards)=81 cloudClient=81

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=82 sum(shards)=81 cloudClient=81
	at __randomizedtesting.SeedInfo.seed([1C3319F285F0B101:9DD597EAF2AFD13D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1280)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1206)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:220)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 10975 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/init-core-data-001
   [junit4]   2> 649787 T2435 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 649795 T2435 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 649796 T2435 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 649796 T2436 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 649897 T2435 oasc.ZkTestServer.run start zk server on port:65317
   [junit4]   2> 649898 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 649939 T2442 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6892ffb5 name:ZooKeeperConnection Watcher:127.0.0.1:65317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 649940 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 649941 T2435 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 649981 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 649983 T2444 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68ba9ea4 name:ZooKeeperConnection Watcher:127.0.0.1:65317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 649984 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 649984 T2435 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 649996 T2435 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 649998 T2435 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 650002 T2435 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 650065 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 650065 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 650095 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 650096 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 650234 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 650235 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 650238 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 650239 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 650276 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 650277 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 650279 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 650280 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 650287 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 650287 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 650289 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 650290 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 650292 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 650293 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 650301 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 650301 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 650304 T2435 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 650304 T2435 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 651270 T2435 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 651274 T2435 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:65338
   [junit4]   2> 651274 T2435 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 651275 T2435 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 651275 T2435 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-002
   [junit4]   2> 651275 T2435 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-002/'
   [junit4]   2> 651312 T2435 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-002/solr.xml
   [junit4]   2> 651387 T2435 oasc.CoreContainer.<init> New CoreContainer 226744590
   [junit4]   2> 651387 T2435 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-002/]
   [junit4]   2> 651389 T2435 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 651389 T2435 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 651389 T2435 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 651390 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 651390 T2435 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 651390 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 651391 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 651391 T2435 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 651391 T2435 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 651392 T2435 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 651392 T2435 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 651393 T2435 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 651393 T2435 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65317/solr
   [junit4]   2> 651393 T2435 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 651394 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 651396 T2455 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b1f2d10 name:ZooKeeperConnection Watcher:127.0.0.1:65317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 651397 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 651398 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 651406 T2457 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ba66663 name:ZooKeeperConnection Watcher:127.0.0.1:65317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 651407 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 651408 T2435 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 651412 T2435 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 651423 T2435 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 651426 T2435 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 651429 T2435 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 651433 T2435 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 651435 T2435 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65338_
   [junit4]   2> 651436 T2435 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65338_
   [junit4]   2> 651445 T2435 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 651447 T2435 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 651460 T2435 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:65338_
   [junit4]   2> 651460 T2435 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 651462 T2435 oasc.Overseer.start Overseer (id=91581765644124163-127.0.0.1:65338_-n_0000000000) starting
   [junit4]   2> 651465 T2435 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 651490 T2459 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 651491 T2435 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 651491 T2459 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:65338_
   [junit4]   2> 651494 T2435 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 651495 T2435 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 651498 T2458 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 651505 T2460 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 651505 T2460 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 651507 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 651507 T2460 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 651508 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 651508 T2458 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:65338",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65338_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 651509 T2458 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 651509 T2458 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 651512 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 651519 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 652508 T2460 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 652508 T2460 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 652509 T2460 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 652509 T2460 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 652511 T2460 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 652511 T2460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-002/collection1/'
   [junit4]   2> 652513 T2460 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 652514 T2460 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 652515 T2460 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 652598 T2460 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_49
   [junit4]   2> 652683 T2460 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 652784 T2460 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 652798 T2460 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 653497 T2460 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 653501 T2460 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 653504 T2460 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 653511 T2460 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 653540 T2460 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 653541 T2460 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 653608 T2460 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/control/data/
   [junit4]   2> 653608 T2460 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a8a7716
   [junit4]   2> 653610 T2460 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/control/data
   [junit4]   2> 653611 T2460 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/control/data/index/
   [junit4]   2> 653613 T2460 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 653614 T2460 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/control/data/index
   [junit4]   2> 653615 T2460 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=91.66015625, floorSegmentMB=0.2275390625, forceMergeDeletesPctAllowed=16.64426317779265, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8286612942458071
   [junit4]   2> 653620 T2460 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 653620 T2460 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 653625 T2460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 653626 T2460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 653626 T2460 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 653626 T2460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 653627 T2460 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 653627 T2460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 653628 T2460 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 653628 T2460 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 653628 T2460 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 653629 T2460 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 653630 T2460 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 653630 T2460 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 653630 T2460 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 653631 T2460 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 653631 T2460 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 653632 T2460 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 653651 T2460 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 653656 T2460 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 653656 T2460 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 653657 T2460 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=33.1630859375, floorSegmentMB=1.703125, forceMergeDeletesPctAllowed=15.45803379378777, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5313125421663342
   [junit4]   2> 653660 T2460 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 653660 T2460 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 653660 T2460 oass.SolrIndexSearcher.<init> Opening Searcher@7f533131[collection1] main
   [junit4]   2> 653661 T2460 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 653662 T2460 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 653662 T2460 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 653663 T2460 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 653663 T2460 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 653664 T2460 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 653664 T2460 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 653665 T2460 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 653665 T2460 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 653665 T2460 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 653669 T2461 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7f533131[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 653672 T2460 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 653672 T2464 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:65338 collection:control_collection shard:shard1
   [junit4]   2> 653673 T2435 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 653673 T2464 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 653673 T2435 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 653675 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 653677 T2466 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e054b3d name:ZooKeeperConnection Watcher:127.0.0.1:65317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 653677 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 653685 T2435 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 653694 T2464 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 653694 T2435 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 653705 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 653705 T2464 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 653705 T2464 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C368 name=collection1 org.apache.solr.core.SolrCore@2d201120 url=http://127.0.0.1:65338/collection1 node=127.0.0.1:65338_ C368_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:65338, core=collection1, node_name=127.0.0.1:65338_}
   [junit4]   2> 653706 T2464 C368 P65338 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:65338/collection1/
   [junit4]   2> 653706 T2464 C368 P65338 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 653706 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 653706 T2464 C368 P65338 oasc.SyncStrategy.syncToMe http://127.0.0.1:65338/collection1/ has no replicas
   [junit4]   2> 653707 T2464 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:65338/collection1/ shard1
   [junit4]   2> 653707 T2464 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 653724 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 653733 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 653734 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 653751 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 653755 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 653858 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 653858 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 653903 T2464 oasc.ZkController.register We are http://127.0.0.1:65338/collection1/ and leader is http://127.0.0.1:65338/collection1/
   [junit4]   2> 653903 T2464 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:65338
   [junit4]   2> 653903 T2464 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 653904 T2464 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 653904 T2464 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 653905 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 653906 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 653906 T2464 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 653906 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 653907 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 653908 T2458 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:65338",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65338_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 653911 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 654015 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 654028 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 654797 T2435 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 654798 T2435 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 654801 T2435 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:65403
   [junit4]   2> 654802 T2435 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 654802 T2435 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 654802 T2435 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-003
   [junit4]   2> 654803 T2435 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-003/'
   [junit4]   2> 654837 T2435 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-003/solr.xml
   [junit4]   2> 654926 T2435 oasc.CoreContainer.<init> New CoreContainer 2013999094
   [junit4]   2> 654926 T2435 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-003/]
   [junit4]   2> 654928 T2435 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 654928 T2435 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 654928 T2435 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 654929 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 654929 T2435 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 654929 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 654930 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 654930 T2435 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 654930 T2435 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 654931 T2435 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 654931 T2435 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 654932 T2435 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 654932 T2435 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65317/solr
   [junit4]   2> 654932 T2435 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 654934 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 654943 T2477 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2679fd0c name:ZooKeeperConnection Watcher:127.0.0.1:65317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 654944 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 654951 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 654973 T2479 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4be97df3 name:ZooKeeperConnection Watcher:127.0.0.1:65317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 654973 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 655007 T2435 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 656011 T2435 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65403_
   [junit4]   2> 656026 T2435 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65403_
   [junit4]   2> 656029 T2479 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 656029 T2457 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 656029 T2466 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 656045 T2480 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 656045 T2480 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 656046 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 656046 T2480 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 656047 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 656047 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 656047 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 656048 T2458 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:65403",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65403_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 656048 T2458 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 656048 T2458 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 656057 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 656058 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 656058 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 656058 T2479 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 657048 T2480 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 657048 T2480 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 657049 T2480 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 657049 T2480 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 657050 T2480 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 657051 T2480 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-003/collection1/'
   [junit4]   2> 657053 T2480 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 657054 T2480 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 657055 T2480 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 657139 T2480 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_49
   [junit4]   2> 657222 T2480 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 657323 T2480 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 657332 T2480 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 657810 T2480 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 657814 T2480 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 657815 T2480 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 657820 T2480 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 657836 T2480 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 657837 T2480 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 657837 T2480 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty1/
   [junit4]   2> 657837 T2480 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a8a7716
   [junit4]   2> 657839 T2480 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty1
   [junit4]   2> 657839 T2480 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty1/index/
   [junit4]   2> 657840 T2480 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 657841 T2480 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty1/index
   [junit4]   2> 657841 T2480 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=91.66015625, floorSegmentMB=0.2275390625, forceMergeDeletesPctAllowed=16.64426317779265, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8286612942458071
   [junit4]   2> 657845 T2480 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 657845 T2480 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 657849 T2480 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 657850 T2480 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 657850 T2480 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 657850 T2480 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 657850 T2480 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 657850 T2480 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 657851 T2480 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 657851 T2480 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 657851 T2480 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 657852 T2480 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 657853 T2480 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 657853 T2480 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 657853 T2480 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 657853 T2480 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 657854 T2480 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 657854 T2480 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 657867 T2480 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 657870 T2480 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 657871 T2480 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 657872 T2480 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=33.1630859375, floorSegmentMB=1.703125, forceMergeDeletesPctAllowed=15.45803379378777, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5313125421663342
   [junit4]   2> 657873 T2480 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 657873 T2480 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 657874 T2480 oass.SolrIndexSearcher.<init> Opening Searcher@3ad8600a[collection1] main
   [junit4]   2> 657874 T2480 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 657875 T2480 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 657875 T2480 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 657875 T2480 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 657876 T2480 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 657876 T2480 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 657876 T2480 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 657877 T2480 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 657877 T2480 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 657877 T2480 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 657881 T2481 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ad8600a[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 657883 T2480 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 657884 T2484 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:65403 collection:collection1 shard:shard2
   [junit4]   2> 657884 T2435 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 657885 T2435 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 657885 T2484 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 657900 T2484 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 657909 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 657909 T2484 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 657909 T2484 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C369 name=collection1 org.apache.solr.core.SolrCore@4d1d67c3 url=http://127.0.0.1:65403/collection1 node=127.0.0.1:65403_ C369_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:65403, core=collection1, node_name=127.0.0.1:65403_}
   [junit4]   2> 657910 T2484 C369 P65403 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:65403/collection1/
   [junit4]   2> 657910 T2484 C369 P65403 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 657910 T2484 C369 P65403 oasc.SyncStrategy.syncToMe http://127.0.0.1:65403/collection1/ has no replicas
   [junit4]   2> 657910 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 657910 T2484 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:65403/collection1/ shard2
   [junit4]   2> 657911 T2484 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 657920 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 657928 T2479 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 657928 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 657928 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 657941 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 657945 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 658048 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 658048 T2479 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 658048 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 658093 T2484 oasc.ZkController.register We are http://127.0.0.1:65403/collection1/ and leader is http://127.0.0.1:65403/collection1/
   [junit4]   2> 658093 T2484 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:65403
   [junit4]   2> 658093 T2484 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 658093 T2484 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 658094 T2484 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 658121 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 658122 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 658122 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 658122 T2484 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 658123 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 658124 T2458 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:65403",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65403_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 658167 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 658276 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 658276 T2479 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 658276 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 659152 T2435 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 659154 T2435 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 659157 T2435 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:65406
   [junit4]   2> 659157 T2435 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 659158 T2435 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 659158 T2435 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-004
   [junit4]   2> 659158 T2435 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-004/'
   [junit4]   2> 659194 T2435 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-004/solr.xml
   [junit4]   2> 659282 T2435 oasc.CoreContainer.<init> New CoreContainer 784187131
   [junit4]   2> 659282 T2435 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-004/]
   [junit4]   2> 659284 T2435 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 659285 T2435 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 659285 T2435 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 659285 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 659286 T2435 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 659286 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 659286 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 659287 T2435 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 659287 T2435 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 659287 T2435 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 659288 T2435 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 659288 T2435 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 659289 T2435 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65317/solr
   [junit4]   2> 659289 T2435 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 659290 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 659326 T2495 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b4568b6 name:ZooKeeperConnection Watcher:127.0.0.1:65317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 659327 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 659330 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 659333 T2497 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@708d5079 name:ZooKeeperConnection Watcher:127.0.0.1:65317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 659333 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 659356 T2435 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 660359 T2435 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65406_
   [junit4]   2> 660369 T2435 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65406_
   [junit4]   2> 660375 T2479 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 660375 T2466 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 660375 T2497 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 660375 T2457 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 660387 T2498 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 660387 T2498 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 660389 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 660389 T2498 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 660389 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 660390 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 660390 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 660391 T2458 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:65406",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65406_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 660391 T2458 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 660391 T2458 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 660400 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 660401 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 660401 T2497 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 660401 T2479 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 660401 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 661390 T2498 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 661390 T2498 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 661391 T2498 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 661391 T2498 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 661392 T2498 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 661392 T2498 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-004/collection1/'
   [junit4]   2> 661395 T2498 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 661396 T2498 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 661396 T2498 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 661483 T2498 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_49
   [junit4]   2> 661574 T2498 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 661675 T2498 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 661689 T2498 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 662437 T2498 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 662442 T2498 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 662445 T2498 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 662451 T2498 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 662476 T2498 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 662476 T2498 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 662477 T2498 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty2/
   [junit4]   2> 662477 T2498 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a8a7716
   [junit4]   2> 662479 T2498 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty2
   [junit4]   2> 662480 T2498 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty2/index/
   [junit4]   2> 662481 T2498 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 662483 T2498 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty2/index
   [junit4]   2> 662484 T2498 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=91.66015625, floorSegmentMB=0.2275390625, forceMergeDeletesPctAllowed=16.64426317779265, segmentsPerTier=19.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8286612942458071
   [junit4]   2> 662489 T2498 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 662489 T2498 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 662496 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 662497 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 662497 T2498 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 662497 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 662498 T2498 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 662498 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 662499 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 662500 T2498 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 662500 T2498 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 662501 T2498 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 662502 T2498 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 662502 T2498 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 662502 T2498 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 662503 T2498 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 662504 T2498 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 662505 T2498 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 662528 T2498 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 662534 T2498 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 662534 T2498 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 662536 T2498 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=33.1630859375, floorSegmentMB=1.703125, forceMergeDeletesPctAllowed=15.45803379378777, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5313125421663342
   [junit4]   2> 662538 T2498 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 662538 T2498 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 662539 T2498 oass.SolrIndexSearcher.<init> Opening Searcher@684fbf57[collection1] main
   [junit4]   2> 662539 T2498 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 662540 T2498 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 662540 T2498 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 662541 T2498 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 662541 T2498 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 662541 T2498 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 662542 T2498 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 662542 T2498 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 662543 T2498 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 662543 T2498 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 662548 T2499 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@684fbf57[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 662551 T2498 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 662552 T2502 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:65406 collection:collection1 shard:shard1
   [junit4]   2> 662553 T2435 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0
   [junit4]   2> 662553 T2502 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 662553 T2435 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 662569 T2502 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 662579 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 662580 T2502 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 662581 T2502 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 662581 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C370 name=collection1 org.apache.solr.core.SolrCore@5746f98f url=http://127.0.0.1:65406/collection1 node=127.0.0.1:65406_ C370_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:65406, core=collection1, node_name=127.0.0.1:65406_}
   [junit4]   2> 662581 T2502 C370 P65406 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:65406/collection1/
   [junit4]   2> 662582 T2502 C370 P65406 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 662582 T2502 C370 P65406 oasc.SyncStrategy.syncToMe http://127.0.0.1:65406/collection1/ has no replicas
   [junit4]   2> 662582 T2502 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:65406/collection1/ shard1
   [junit4]   2> 662583 T2502 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 662585 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 662594 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662594 T2497 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662594 T2479 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662594 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662605 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 662608 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 662710 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662710 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662710 T2497 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662710 T2479 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662760 T2502 oasc.ZkController.register We are http://127.0.0.1:65406/collection1/ and leader is http://127.0.0.1:65406/collection1/
   [junit4]   2> 662761 T2502 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:65406
   [junit4]   2> 662761 T2502 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 662761 T2502 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 662761 T2502 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 662768 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 662768 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 662768 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 662770 T2458 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 662770 T2502 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 662771 T2458 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:65406",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:65406_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 662774 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 662877 T2497 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662878 T2457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662878 T2479 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 662877 T2466 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 663663 T2435 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 663664 T2435 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 663668 T2435 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:65410
   [junit4]   2> 663669 T2435 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 663669 T2435 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 663669 T2435 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-005
   [junit4]   2> 663670 T2435 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-005/'
   [junit4]   2> 663706 T2435 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-005/solr.xml
   [junit4]   2> 663801 T2435 oasc.CoreContainer.<init> New CoreContainer 1829700182
   [junit4]   2> 663802 T2435 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-005/]
   [junit4]   2> 663803 T2435 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 663804 T2435 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 663804 T2435 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 663804 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 663805 T2435 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 663805 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 663805 T2435 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 663806 T2435 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 663806 T2435 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 663807 T2435 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 663807 T2435 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 663807 T2435 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 663808 T2435 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:65317/solr
   [junit4]   2> 663808 T2435 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 663809 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 663812 T2513 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74629409 name:ZooKeeperConnection Watcher:127.0.0.1:65317 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 663813 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 663816 T2435 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 663831 T2515 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6894beef name:ZooKeeperConnection Watcher:127.0.0.1:65317/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 663832 T2435 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 663849 T2435 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 664852 T2435 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65410_
   [junit4]   2> 664867 T2435 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65410_
   [junit4]   2> 664873 T2479 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 664873 T2515 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 664873 T2466 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 664873 T2497 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 664873 T2457 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 664884 T2516 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 664884 T2516 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 664886 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 664886 T2516 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 664886 T2457 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 664886 T2457 oasc.DistributedQueu

[...truncated too long message...]

e/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty6
   [junit4]   2> 856672 T2577 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 856672 T2577 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 856714 T2598 C393 P65460 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 856715 T2598 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
   [junit4]   2> 856715 T2598 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@ca8490e
   [junit4]   2> 856732 T2598 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=68,cumulative_deletesById=31,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=7873,transaction_logs_total_number=1}
   [junit4]   2> 856732 T2598 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 856733 T2598 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 856733 T2598 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 856734 T2598 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 856735 T2598 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 856735 T2598 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty7/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty7/index;done=false>>]
   [junit4]   2> 856735 T2598 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty7/index
   [junit4]   2> 856735 T2598 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty7 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty7;done=false>>]
   [junit4]   2> 856736 T2598 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001/tempDir-001/jetty7
   [junit4]   2> 856736 T2598 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 856736 T2598 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeyNothingIsSafeTest-1C3319F285F0B101-001
   [junit4]   2> NOTE: test params are: codec=Lucene46: {a_t=PostingsFormat(name=FSTPulsing41), id=PostingsFormat(name=MockRandom), rnd_b=PostingsFormat(name=MockRandom), _version_=PostingsFormat(name=FSTPulsing41), a_si=PostingsFormat(name=MockSep)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=fr_FR, timezone=Asia/Riyadh88
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=313831592,total=522190848
   [junit4]   2> NOTE: All tests run in this JVM: [PolyFieldTest, EnumFieldTest, TestManagedResource, NumericFieldsTest, MigrateRouteKeyTest, RecoveryZkTest, TestHashPartitioner, CoreMergeIndexesAdminHandlerTest, SliceStateTest, UniqFieldsUpdateProcessorFactoryTest, DateMathParserTest, CoreContainerCoreInitFailuresTest, AnalysisErrorHandlingTest, HdfsLockFactoryTest, TestFileDictionaryLookup, TestSolrQueryParserDefaultOperatorResource, TestDocSet, QueryEqualityTest, StressHdfsTest, TestQuerySenderListener, TestSort, JsonLoaderTest, RequestHandlersTest, TestExceedMaxTermLength, TestCollationKeyRangeQueries, ResourceLoaderTest, RequiredFieldsTest, ZkSolrClientTest, LeaderElectionTest, SOLR749Test, ReturnFieldsTest, DistributedQueryComponentOptimizationTest, TestSurroundQueryParser, TestCollationFieldDocValues, TestDistributedGrouping, ConnectionManagerTest, TestRangeQuery, SortByFunctionTest, CustomCollectionTest, SampleTest, TestOverriddenPrefixQueryForCustomFieldType, TestShortCircuitedRequests, ChaosMonkeySafeLeaderTest, LegacyHTMLStripCharFilterTest, ChangedSchemaMergeTest, LukeRequestHandlerTest, IndexSchemaTest, DistanceFunctionTest, BadComponentTest, PreAnalyzedUpdateProcessorTest, TestIndexSearcher, PathHierarchyTokenizerFactoryTest, TestGroupingSearch, DirectSolrConnectionTest, TestAddFieldRealTimeGet, ResponseLogComponentTest, DirectSolrSpellCheckerTest, SolrPluginUtilsTest, TestDocBasedVersionConstraints, HdfsRecoveryZkTest, XsltUpdateRequestHandlerTest, TestSchemaNameResource, HdfsBasicDistributedZkTest, DistribDocExpirationUpdateProcessorTest, TestMultiCoreConfBootstrap, DistributedExpandComponentTest, DocumentBuilderTest, TestSolrXml, SolrRequestParserTest, TriLevelCompositeIdRoutingTest, CSVRequestHandlerTest, TestSchemaSimilarityResource, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed on J0 in 207.83s, 1 test, 1 failure <<< FAILURES!

[...truncated 681 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:467: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:1282: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java7/lucene/common-build.xml:908: There were test failures: 387 suites, 1619 tests, 1 failure, 37 ignored (16 assumptions)

Total time: 75 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message