lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 884 - Failure!
Date Sun, 13 Oct 2013 01:37:23 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/884/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
IOException occured when talking to server at: http://127.0.0.1:52951/b_nc/l/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:52951/b_nc/l/collection1
	at __randomizedtesting.SeedInfo.seed([205FEE4FD7D58062:A1B96057A08AE05E]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:476)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1161)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:194)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.SocketException: Invalid argument
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:366)
	... 46 more




Build Log:
[...truncated 10098 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> 1155596 T2745 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /b_nc/l
   [junit4]   2> 1155603 T2745 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyNothingIsSafeTest-1381625434919
   [junit4]   2> 1155605 T2745 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1155606 T2746 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1155706 T2745 oasc.ZkTestServer.run start zk server on port:52915
   [junit4]   2> 1155713 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1155719 T2752 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7791209b name:ZooKeeperConnection Watcher:127.0.0.1:52915 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1155720 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1155721 T2745 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1155731 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1155735 T2754 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3603093a name:ZooKeeperConnection Watcher:127.0.0.1:52915/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1155736 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1155736 T2745 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1155743 T2745 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1155749 T2745 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1155755 T2745 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1155761 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1155762 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1155773 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1155774 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1155781 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1155782 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1155790 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1155791 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1155798 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1155798 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1155805 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1155806 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1155813 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1155814 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1155822 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1155823 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1155829 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1155830 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1155837 T2745 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1155838 T2745 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1156263 T2745 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1156285 T2745 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52918
   [junit4]   2> 1156286 T2745 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1156286 T2745 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1156287 T2745 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1381625435161
   [junit4]   2> 1156287 T2745 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1381625435161/'
   [junit4]   2> 1156332 T2745 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1381625435161/solr.xml
   [junit4]   2> 1156450 T2745 oasc.CoreContainer.<init> New CoreContainer 122335889
   [junit4]   2> 1156451 T2745 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1381625435161/]
   [junit4]   2> 1156453 T2745 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1156454 T2745 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1156455 T2745 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1156455 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1156455 T2745 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1156456 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1156456 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1156457 T2745 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1156457 T2745 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1156457 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 1156462 T2745 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1156463 T2745 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1156463 T2745 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52915/solr
   [junit4]   2> 1156464 T2745 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1156465 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1156470 T2765 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69744579 name:ZooKeeperConnection Watcher:127.0.0.1:52915 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1156471 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1156474 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 1156477 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1156481 T2767 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70203823 name:ZooKeeperConnection Watcher:127.0.0.1:52915/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1156482 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1156485 T2745 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1156493 T2745 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1156501 T2745 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1156505 T2745 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52918_b_nc%2Fl
   [junit4]   2> 1156507 T2745 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52918_b_nc%2Fl
   [junit4]   2> 1156514 T2745 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1156525 T2745 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1156529 T2745 oasc.Overseer.start Overseer (id=90546204503375875-127.0.0.1:52918_b_nc%2Fl-n_0000000000) starting
   [junit4]   2> 1156538 T2745 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1156549 T2769 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1156551 T2745 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1156559 T2745 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1156565 T2745 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1156576 T2768 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1156585 T2770 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1156586 T2770 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1156589 T2770 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1158093 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1158096 T2768 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52918/b_nc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52918_b_nc%2Fl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1158097 T2768 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1158097 T2768 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1158115 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1158592 T2770 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1158593 T2770 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1381625435161/collection1
   [junit4]   2> 1158593 T2770 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1158595 T2770 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1158595 T2770 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1158598 T2770 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1381625435161/collection1/'
   [junit4]   2> 1158601 T2770 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1381625435161/collection1/lib/classes/' to classloader
   [junit4]   2> 1158602 T2770 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1381625435161/collection1/lib/README' to classloader
   [junit4]   2> 1158688 T2770 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
   [junit4]   2> 1158820 T2770 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1158824 T2770 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1158838 T2770 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1159668 T2770 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1159669 T2770 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1159670 T2770 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1159684 T2770 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1159690 T2770 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1159724 T2770 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1159732 T2770 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1159739 T2770 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1159741 T2770 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1159742 T2770 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1159742 T2770 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1159744 T2770 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1159745 T2770 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1159745 T2770 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1159746 T2770 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1381625435161/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/control/data/
   [junit4]   2> 1159746 T2770 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ffe8d15
   [junit4]   2> 1159748 T2770 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/control/data
   [junit4]   2> 1159749 T2770 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/control/data/index/
   [junit4]   2> 1159750 T2770 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1159751 T2770 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/control/data/index
   [junit4]   2> 1159751 T2770 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1159757 T2770 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1159757 T2770 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1159763 T2770 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1159764 T2770 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1159765 T2770 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1159765 T2770 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1159765 T2770 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1159766 T2770 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1159767 T2770 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1159767 T2770 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1159768 T2770 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1159768 T2770 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1159769 T2770 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1159770 T2770 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1159770 T2770 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1159771 T2770 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1159772 T2770 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1159785 T2770 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1159790 T2770 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1159791 T2770 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1159792 T2770 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1159794 T2770 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1159794 T2770 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1159795 T2770 oass.SolrIndexSearcher.<init> Opening Searcher@2013dd7d main
   [junit4]   2> 1159804 T2771 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2013dd7d main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1159806 T2770 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1159806 T2770 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52918/b_nc/l collection:control_collection shard:shard1
   [junit4]   2> 1159807 T2770 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 1159810 T2770 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1159827 T2770 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1159833 T2770 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1159833 T2770 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1159833 T2770 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52918/b_nc/l/collection1/
   [junit4]   2> 1159834 T2770 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1159834 T2770 oasc.SyncStrategy.syncToMe http://127.0.0.1:52918/b_nc/l/collection1/ has no replicas
   [junit4]   2> 1159835 T2770 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52918/b_nc/l/collection1/ shard1
   [junit4]   2> 1159835 T2770 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1161144 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1161186 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1161248 T2770 oasc.ZkController.register We are http://127.0.0.1:52918/b_nc/l/collection1/ and leader is http://127.0.0.1:52918/b_nc/l/collection1/
   [junit4]   2> 1161250 T2770 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52918/b_nc/l
   [junit4]   2> 1161251 T2770 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1161252 T2770 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1161253 T2770 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1161261 T2770 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1161268 T2745 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1161269 T2745 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1161271 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 1161276 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 1161282 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1161293 T2774 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f2f1c45 name:ZooKeeperConnection Watcher:127.0.0.1:52915/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1161294 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1161301 T2745 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1161318 T2745 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 1161746 T2745 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1161752 T2745 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52922
   [junit4]   2> 1161754 T2745 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1161754 T2745 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1161755 T2745 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1381625440634
   [junit4]   2> 1161756 T2745 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1381625440634/'
   [junit4]   2> 1161799 T2745 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1381625440634/solr.xml
   [junit4]   2> 1161907 T2745 oasc.CoreContainer.<init> New CoreContainer 891108243
   [junit4]   2> 1161908 T2745 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1381625440634/]
   [junit4]   2> 1161911 T2745 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1161911 T2745 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1161912 T2745 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1161912 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1161913 T2745 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1161913 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1161914 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1161914 T2745 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1161915 T2745 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1161915 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 1161921 T2745 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1161922 T2745 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1161923 T2745 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52915/solr
   [junit4]   2> 1161923 T2745 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1161926 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1161932 T2785 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@80567ce name:ZooKeeperConnection Watcher:127.0.0.1:52915 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1161934 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1161939 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 1161942 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1161948 T2787 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@559f6e8a name:ZooKeeperConnection Watcher:127.0.0.1:52915/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1161948 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1161958 T2745 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1162723 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1162727 T2768 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52918/b_nc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52918_b_nc%2Fl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1162749 T2774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1162750 T2787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1162750 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1162978 T2745 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52922_b_nc%2Fl
   [junit4]   2> 1162984 T2745 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52922_b_nc%2Fl
   [junit4]   2> 1163000 T2774 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1163002 T2787 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1163002 T2767 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1163035 T2788 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1163035 T2788 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1163038 T2788 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1164279 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1164283 T2768 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52922/b_nc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52922_b_nc%2Fl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1164284 T2768 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1164285 T2768 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1164308 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1164309 T2774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1164309 T2787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1165042 T2788 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1165043 T2788 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1381625440634/collection1
   [junit4]   2> 1165044 T2788 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1165048 T2788 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1165048 T2788 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1165053 T2788 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1381625440634/collection1/'
   [junit4]   2> 1165054 T2788 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1381625440634/collection1/lib/classes/' to classloader
   [junit4]   2> 1165055 T2788 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1381625440634/collection1/lib/README' to classloader
   [junit4]   2> 1165142 T2788 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
   [junit4]   2> 1165228 T2788 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1165231 T2788 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1165243 T2788 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1166083 T2788 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1166084 T2788 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1166085 T2788 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1166098 T2788 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1166102 T2788 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1166131 T2788 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1166138 T2788 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1166145 T2788 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1166147 T2788 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1166147 T2788 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1166148 T2788 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1166150 T2788 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1166150 T2788 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1166151 T2788 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1166151 T2788 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1381625440634/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty1/
   [junit4]   2> 1166152 T2788 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ffe8d15
   [junit4]   2> 1166153 T2788 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty1
   [junit4]   2> 1166154 T2788 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty1/index/
   [junit4]   2> 1166156 T2788 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1166157 T2788 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty1/index
   [junit4]   2> 1166158 T2788 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1166163 T2788 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1166163 T2788 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1166173 T2788 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1166175 T2788 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1166176 T2788 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1166177 T2788 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1166177 T2788 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1166178 T2788 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1166178 T2788 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1166179 T2788 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1166179 T2788 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1166180 T2788 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1166181 T2788 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1166182 T2788 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1166182 T2788 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1166183 T2788 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1166184 T2788 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1166197 T2788 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1166202 T2788 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1166202 T2788 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1166204 T2788 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1166206 T2788 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1166206 T2788 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1166207 T2788 oass.SolrIndexSearcher.<init> Opening Searcher@4afcd2a9 main
   [junit4]   2> 1166216 T2789 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4afcd2a9 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1166219 T2788 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1166220 T2788 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52922/b_nc/l collection:collection1 shard:shard1
   [junit4]   2> 1166220 T2788 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 1166223 T2788 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1166240 T2788 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1166246 T2788 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1166246 T2788 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1166246 T2788 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52922/b_nc/l/collection1/
   [junit4]   2> 1166247 T2788 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1166247 T2788 oasc.SyncStrategy.syncToMe http://127.0.0.1:52922/b_nc/l/collection1/ has no replicas
   [junit4]   2> 1166247 T2788 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52922/b_nc/l/collection1/ shard1
   [junit4]   2> 1166248 T2788 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1167335 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1167355 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1167356 T2787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1167356 T2774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1167403 T2788 oasc.ZkController.register We are http://127.0.0.1:52922/b_nc/l/collection1/ and leader is http://127.0.0.1:52922/b_nc/l/collection1/
   [junit4]   2> 1167404 T2788 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52922/b_nc/l
   [junit4]   2> 1167404 T2788 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1167404 T2788 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1167405 T2788 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1167408 T2788 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1167411 T2745 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1167412 T2745 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1167412 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 1167909 T2745 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1167915 T2745 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52925
   [junit4]   2> 1167916 T2745 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1167917 T2745 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1167917 T2745 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1381625446729
   [junit4]   2> 1167918 T2745 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1381625446729/'
   [junit4]   2> 1167963 T2745 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1381625446729/solr.xml
   [junit4]   2> 1168091 T2745 oasc.CoreContainer.<init> New CoreContainer 362751340
   [junit4]   2> 1168092 T2745 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1381625446729/]
   [junit4]   2> 1168094 T2745 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1168095 T2745 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1168095 T2745 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1168096 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1168096 T2745 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1168097 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1168097 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1168098 T2745 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1168098 T2745 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1168099 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 1168105 T2745 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1168106 T2745 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1168106 T2745 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52915/solr
   [junit4]   2> 1168107 T2745 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1168109 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1168113 T2801 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1195ab57 name:ZooKeeperConnection Watcher:127.0.0.1:52915 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1168115 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1168118 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 1168121 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1168125 T2803 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35d1c9ef name:ZooKeeperConnection Watcher:127.0.0.1:52915/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1168125 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1168135 T2745 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1168876 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1168881 T2768 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52922/b_nc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52922_b_nc%2Fl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1168903 T2803 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1168904 T2774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1168903 T2787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1168905 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1169147 T2745 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52925_b_nc%2Fl
   [junit4]   2> 1169153 T2745 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52925_b_nc%2Fl
   [junit4]   2> 1169170 T2803 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1169172 T2787 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1169173 T2767 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1169174 T2774 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1169212 T2804 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1169212 T2804 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1169215 T2804 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1170432 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1170437 T2768 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52925/b_nc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52925_b_nc%2Fl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1170438 T2768 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1170439 T2768 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1170460 T2787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1170462 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1170463 T2803 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1170462 T2774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1171220 T2804 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1171220 T2804 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1381625446729/collection1
   [junit4]   2> 1171221 T2804 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1171223 T2804 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1171223 T2804 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1171227 T2804 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1381625446729/collection1/'
   [junit4]   2> 1171229 T2804 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1381625446729/collection1/lib/classes/' to classloader
   [junit4]   2> 1171230 T2804 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1381625446729/collection1/lib/README' to classloader
   [junit4]   2> 1171324 T2804 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
   [junit4]   2> 1171442 T2804 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1171445 T2804 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1171461 T2804 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1172306 T2804 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1172307 T2804 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1172308 T2804 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1172320 T2804 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1172326 T2804 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1172358 T2804 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1172366 T2804 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1172375 T2804 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1172379 T2804 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1172380 T2804 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1172380 T2804 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1172384 T2804 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1172385 T2804 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1172386 T2804 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1172387 T2804 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1381625446729/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty2/
   [junit4]   2> 1172387 T2804 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ffe8d15
   [junit4]   2> 1172390 T2804 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty2
   [junit4]   2> 1172392 T2804 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty2/index/
   [junit4]   2> 1172393 T2804 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1172394 T2804 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty2/index
   [junit4]   2> 1172396 T2804 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1172401 T2804 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1172402 T2804 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1172411 T2804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1172412 T2804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1172412 T2804 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1172413 T2804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1172414 T2804 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1172414 T2804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1172415 T2804 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1172416 T2804 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1172416 T2804 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1172418 T2804 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1172419 T2804 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1172420 T2804 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1172420 T2804 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1172421 T2804 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1172422 T2804 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1172441 T2804 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1172448 T2804 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1172448 T2804 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1172449 T2804 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1172452 T2804 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1172452 T2804 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1172453 T2804 oass.SolrIndexSearcher.<init> Opening Searcher@4c14fe71 main
   [junit4]   2> 1172465 T2805 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c14fe71 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 1172469 T2804 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1172470 T2804 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52925/b_nc/l collection:collection1 shard:shard2
   [junit4]   2> 1172470 T2804 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 1172473 T2804 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1172490 T2804 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1172497 T2804 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1172497 T2804 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1172498 T2804 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52925/b_nc/l/collection1/
   [junit4]   2> 1172498 T2804 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1172498 T2804 oasc.SyncStrategy.syncToMe http://127.0.0.1:52925/b_nc/l/collection1/ has no replicas
   [junit4]   2> 1172499 T2804 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52925/b_nc/l/collection1/ shard2
   [junit4]   2> 1172499 T2804 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1173494 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1173534 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1173534 T2803 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1173534 T2774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1173534 T2787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1173570 T2804 oasc.ZkController.register We are http://127.0.0.1:52925/b_nc/l/collection1/ and leader is http://127.0.0.1:52925/b_nc/l/collection1/
   [junit4]   2> 1173571 T2804 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52925/b_nc/l
   [junit4]   2> 1173571 T2804 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1173572 T2804 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1173572 T2804 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1173575 T2804 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1173578 T2745 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 1173578 T2745 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1173579 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 1174185 T2745 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1174192 T2745 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52928
   [junit4]   2> 1174194 T2745 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1174195 T2745 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1174196 T2745 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1381625452895
   [junit4]   2> 1174196 T2745 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1381625452895/'
   [junit4]   2> 1174262 T2745 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1381625452895/solr.xml
   [junit4]   2> 1174403 T2745 oasc.CoreContainer.<init> New CoreContainer 1186152334
   [junit4]   2> 1174404 T2745 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1381625452895/]
   [junit4]   2> 1174407 T2745 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1174408 T2745 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1174408 T2745 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1174409 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1174410 T2745 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1174412 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1174413 T2745 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1174414 T2745 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1174415 T2745 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1174416 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 1174423 T2745 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1174423 T2745 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1174425 T2745 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52915/solr
   [junit4]   2> 1174425 T2745 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1174427 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1174432 T2817 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@40922ac6 name:ZooKeeperConnection Watcher:127.0.0.1:52915 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1174433 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1174438 T2745 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 1174442 T2745 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1174445 T2819 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4bf65009 name:ZooKeeperConnection Watcher:127.0.0.1:52915/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1174446 T2745 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1174458 T2745 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1175062 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1175064 T2768 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52925/b_nc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52925_b_nc%2Fl",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1175079 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1175080 T2774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1175079 T2803 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1175080 T2787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1175080 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1175470 T2745 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52928_b_nc%2Fl
   [junit4]   2> 1175474 T2745 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52928_b_nc%2Fl
   [junit4]   2> 1175484 T2774 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1175485 T2787 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1175486 T2767 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1175487 T2803 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1175486 T2819 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 1175515 T2820 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1175516 T2820 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1175521 T2820 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1176602 T2768 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1176604 T2768 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:52928/b_nc/l",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:52928_b_nc%2Fl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node3"}
   [junit4]   2> 1176605 T2768 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1176605 T2768 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1176621 T2767 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1176622 T2787 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1176622 T2819 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1176622 T2803 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1176622 T2774 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 1177523 T2820 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1177524 T2820 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1381625452895/collection1
   [junit4]   2> 1177524 T2820 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1177526 T2820 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1177527 T2820 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1177531 T2820 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1381625452895/collection1/'
   [junit4]   2> 1177534 T2820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1381625452895/collection1/lib/classes/' to classloader
   [junit4]   2> 1177535 T2820 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1381625452895/collection1/lib/README' to classloader
   [junit4]   2> 1177672 T2820 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
   [junit4]   2> 1177829 T2820 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1177834 T2820 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1177858 T2820 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1179252 T2820 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1179255 T2820 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 1179257 T2820 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1179276 T2820 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1179283 T2820 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1179339 T2820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1179364 T2820 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1179373 T2820 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1179376 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1179376 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1179377 T2820 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1179382 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1179383 T2820 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1179384 T2820 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1179385 T2820 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1381625452895/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty3/
   [junit4]   2> 1179385 T2820 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ffe8d15
   [junit4]   2> 1179387 T2820 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty3
   [junit4]   2> 1179389 T2820 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty3/index/
   [junit4]   2> 1179391 T2820 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 1179393 T2820 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty3/index
   [junit4]   2> 1179395 T2820 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=28, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1179400 T2820 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1381625434919/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 1179400 T2820 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1179409 T2820 oasup.UpdateRequestProcessorChain.init cre

[...truncated too long message...]

tions/control_collection/shards (0)
   [junit4]   1>     /solr/collections/control_collection/leader_elect (1)
   [junit4]   1>      /solr/collections/control_collection/leader_elect/shard1 (1)
   [junit4]   1>       /solr/collections/control_collection/leader_elect/shard1/election (1)
   [junit4]   1>        /solr/collections/control_collection/leader_elect/shard1/election/90546204503375875-core_node1-n_0000000000 (0)
   [junit4]   1>     /solr/collections/control_collection/leaders (1)
   [junit4]   1>      /solr/collections/control_collection/leaders/shard1 (0)
   [junit4]   1>      DATA:
   [junit4]   1>          {
   [junit4]   1>            "core":"collection1",
   [junit4]   1>            "node_name":"127.0.0.1:52918_b_nc%2Fl",
   [junit4]   1>            "base_url":"http://127.0.0.1:52918/b_nc/l"}
   [junit4]   1>   /solr/overseer_elect (2)
   [junit4]   1>    /solr/overseer_elect/election (8)
   [junit4]   1>     /solr/overseer_elect/election/90546204503375884-127.0.0.1:52932_b_nc%2Fl-n_0000000004 (0)
   [junit4]   1>     /solr/overseer_elect/election/90546204503375882-127.0.0.1:52928_b_nc%2Fl-n_0000000003 (0)
   [junit4]   1>     /solr/overseer_elect/election/90546204503375888-127.0.0.1:52946_b_nc%2Fl-n_0000000006 (0)
   [junit4]   1>     /solr/overseer_elect/election/90546204503375886-127.0.0.1:52939_b_nc%2Fl-n_0000000005 (0)
   [junit4]   1>     /solr/overseer_elect/election/90546204503375880-127.0.0.1:52925_b_nc%2Fl-n_0000000002 (0)
   [junit4]   1>     /solr/overseer_elect/election/90546204503375878-127.0.0.1:52922_b_nc%2Fl-n_0000000001 (0)
   [junit4]   1>     /solr/overseer_elect/election/90546204503375890-127.0.0.1:52951_b_nc%2Fl-n_0000000007 (0)
   [junit4]   1>     /solr/overseer_elect/election/90546204503375875-127.0.0.1:52918_b_nc%2Fl-n_0000000000 (0)
   [junit4]   1>    /solr/overseer_elect/leader (0)
   [junit4]   1>    DATA:
   [junit4]   1>        {"id":"90546204503375875-127.0.0.1:52918_b_nc%2Fl-n_0000000000"}
   [junit4]   1>  /zookeeper (1)
   [junit4]   1>  DATA:
   [junit4]   1>      
   [junit4]   1> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=testDistribSearch -Dtests.seed=205FEE4FD7D58062 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=PNT -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    160s | ChaosMonkeyNothingIsSafeTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:52951/b_nc/l/collection1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([205FEE4FD7D58062:A1B96057A08AE05E]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:476)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
   [junit4]    > 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1161)
   [junit4]    > 	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:194)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
   [junit4]    > Caused by: java.net.SocketException: Invalid argument
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
   [junit4]    > 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:366)
   [junit4]    > 	... 46 more
   [junit4]   2> 1315198 T2745 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 159609 T2744 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 1315671 T2890 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene45: {timestamp=PostingsFormat(name=MockSep), range_facet_l=PostingsFormat(name=MockSep), id=PostingsFormat(name=MockRandom), range_facet_sl=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_t=PostingsFormat(name=MockSep), text=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_si=Lucene41(blocksize=128), _version_=PostingsFormat(name=MockSep), other_tl1=PostingsFormat(name=MockSep), multiDefault=Lucene41(blocksize=128), a_si=Lucene41(blocksize=128), intDefault=PostingsFormat(name=MockSep)}, docValues:{}, sim=DefaultSimilarity, locale=mk, timezone=PNT
   [junit4]   2> NOTE: Mac OS X 10.8.4 x86_64/Oracle Corporation 1.7.0_40 (64-bit)/cpus=2,threads=1,free=174675520,total=494927872
   [junit4]   2> NOTE: All tests run in this JVM: [AddBlockUpdateTest, HardAutoCommitTest, LeaderElectionTest, StatsComponentTest, TestSort, LegacyHTMLStripCharFilterTest, DirectUpdateHandlerOptimizeTest, TestReload, TestSolrJ, SpellPossibilityIteratorTest, SyncSliceTest, ResponseLogComponentTest, TestCollationField, CoreAdminHandlerTest, CollectionsAPIDistributedZkTest, TestRemoteStreaming, TestSolrXMLSerializer, PeerSyncTest, ShowFileRequestHandlerTest, AliasIntegrationTest, TestQuerySenderNoQuery, JsonLoaderTest, HdfsCollectionsAPIDistributedZkTest, ScriptEngineTest, PluginInfoTest, TestConfig, DirectSolrConnectionTest, ZkCLITest, PreAnalyzedUpdateProcessorTest, TestSolrXml, DefaultValueUpdateProcessorTest, TestSuggestSpellingConverter, TestPostingsSolrHighlighter, SoftAutoCommitTest, AssignTest, HdfsLockFactoryTest, TestDistributedSearch, TestUniqueKeyFieldResource, CircularListTest, TestShardHandlerFactory, TestLMJelinekMercerSimilarityFactory, SolrCoreTest, WordBreakSolrSpellCheckerTest, BlockDirectoryTest, DocumentBuilderTest, TestDFRSimilarityFactory, OverseerTest, TestSchemaResource, TestAtomicUpdateErrorCases, TestUpdate, ChaosMonkeySafeLeaderTest, TestQueryUtils, BadComponentTest, SolrInfoMBeanTest, TestSolrDeletionPolicy1, LeaderElectionIntegrationTest, SuggesterFSTTest, HdfsUnloadDistributedZkTest, TestFuzzyAnalyzedSuggestions, InfoHandlerTest, TestCoreDiscovery, TestJoin, CopyFieldTest, SpellCheckCollatorTest, SolrRequestParserTest, TestNRTOpen, XmlUpdateRequestHandlerTest, TestRecoveryHdfs, TestFieldTypeCollectionResource, TermVectorComponentTest, RequestHandlersTest, DistanceFunctionTest, TestSystemIdResolver, SignatureUpdateProcessorFactoryTest, TestHashPartitioner, TestSolrIndexConfig, TestIndexingPerformance, TestCodecSupport, SOLR749Test, ShardSplitTest, DirectUpdateHandlerTest, ChaosMonkeyNothingIsSafeTest]
   [junit4] Completed in 160.16s, 1 test, 1 error <<< FAILURES!

[...truncated 766 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:428: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:408: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:465: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1256: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:899: There were test failures: 327 suites, 1453 tests, 1 error, 39 ignored (5 assumptions)

Total time: 106 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message