lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_25) - Build # 6252 - Still Failing!
Date Sat, 22 Jun 2013 08:41:01 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6252/
Java: 32bit/jdk1.7.0_25 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 305 from http://127.0.0.1:54846/collection1lastClient and got 247 from http://127.0.0.1:41066/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:54846/collection1lastClient and got 247 from http://127.0.0.1:41066/collection1
	at __randomizedtesting.SeedInfo.seed([CB968363466C6444:4A700D7B31330478]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 9436 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 359979 T1331 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 359985 T1331 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1371889555429
[junit4:junit4]   2> 359985 T1331 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 359986 T1332 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 360086 T1331 oasc.ZkTestServer.run start zk server on port:42857
[junit4:junit4]   2> 360087 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 360230 T1338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8357d0 name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 360231 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 360231 T1331 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 360235 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 360237 T1340 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18aba7c name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 360237 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 360238 T1331 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 360240 T1331 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 360242 T1331 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 360244 T1331 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 360246 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 360247 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 360253 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 360253 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 360256 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 360256 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 360259 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 360259 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 360261 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 360261 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 360264 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 360264 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 360266 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 360266 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 360268 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 360269 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 360271 T1331 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 360271 T1331 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 360364 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 360368 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57702
[junit4:junit4]   2> 360369 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 360369 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 360369 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718
[junit4:junit4]   2> 360370 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/solr.xml
[junit4:junit4]   2> 360370 T1331 oasc.CoreContainer.<init> New CoreContainer 24427082
[junit4:junit4]   2> 360371 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/'
[junit4:junit4]   2> 360371 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/'
[junit4:junit4]   2> 360439 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 360440 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 360440 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 360440 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 360441 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 360441 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 360442 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 360442 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 360442 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 360443 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 360446 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 360447 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4]   2> 360447 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 360448 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 360449 T1351 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b280a name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 360450 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 360451 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 360454 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 360455 T1353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d75942 name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 360455 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 360457 T1331 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 360467 T1331 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 360470 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 360472 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57702_
[junit4:junit4]   2> 360472 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57702_
[junit4:junit4]   2> 360477 T1331 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 360482 T1331 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 360484 T1331 oasc.Overseer.start Overseer (id=89908153904726019-127.0.0.1:57702_-n_0000000000) starting
[junit4:junit4]   2> 360488 T1331 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 360493 T1355 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 360493 T1331 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 360503 T1331 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 360505 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 360508 T1354 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 360510 T1356 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 360510 T1356 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 362012 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 362013 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:57702_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:57702"}
[junit4:junit4]   2> 362013 T1354 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 362013 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 362016 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 362511 T1356 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1
[junit4:junit4]   2> 362511 T1356 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 362512 T1356 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 362512 T1356 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 362514 T1356 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1/'
[junit4:junit4]   2> 362514 T1356 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1/lib/README' to classloader
[junit4:junit4]   2> 362514 T1356 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 362553 T1356 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 362584 T1356 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 362586 T1356 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 362590 T1356 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 362978 T1356 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 362985 T1356 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 362988 T1356 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 363004 T1356 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 363007 T1356 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 363009 T1356 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 363010 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 363010 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 363010 T1356 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 363011 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 363011 T1356 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 363012 T1356 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 363018 T1356 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1371889555718/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/
[junit4:junit4]   2> 363018 T1356 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4]   2> 363018 T1356 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 363018 T1356 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data
[junit4:junit4]   2> 363019 T1356 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index/
[junit4:junit4]   2> 363019 T1356 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 363019 T1356 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index
[junit4:junit4]   2> 363022 T1356 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 363022 T1356 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 363023 T1356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 363024 T1356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 363024 T1356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 363024 T1356 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 363025 T1356 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 363025 T1356 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 363025 T1356 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 363025 T1356 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 363026 T1356 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 363028 T1356 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 363031 T1356 oass.SolrIndexSearcher.<init> Opening Searcher@d4ee71 main
[junit4:junit4]   2> 363032 T1356 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 363032 T1356 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 363035 T1357 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d4ee71 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 363035 T1356 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 363035 T1356 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57702 collection:control_collection shard:shard1
[junit4:junit4]   2> 363036 T1356 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 363041 T1356 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 363043 T1356 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 363044 T1356 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 363044 T1356 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57702/collection1/
[junit4:junit4]   2> 363044 T1356 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 363044 T1356 oasc.SyncStrategy.syncToMe http://127.0.0.1:57702/collection1/ has no replicas
[junit4:junit4]   2> 363044 T1356 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57702/collection1/
[junit4:junit4]   2> 363044 T1356 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 363520 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 363526 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 363550 T1356 oasc.ZkController.register We are http://127.0.0.1:57702/collection1/ and leader is http://127.0.0.1:57702/collection1/
[junit4:junit4]   2> 363550 T1356 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57702
[junit4:junit4]   2> 363551 T1356 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 363551 T1356 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 363551 T1356 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 363553 T1356 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 363554 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 363554 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 363555 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 363559 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 363560 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 363561 T1360 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fe720e name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 363561 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 363563 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 363565 T1331 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 363638 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 363640 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54846
[junit4:junit4]   2> 363640 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 363640 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 363641 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010
[junit4:junit4]   2> 363641 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/solr.xml
[junit4:junit4]   2> 363641 T1331 oasc.CoreContainer.<init> New CoreContainer 14823272
[junit4:junit4]   2> 363642 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/'
[junit4:junit4]   2> 363642 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/'
[junit4:junit4]   2> 363698 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 363698 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 363699 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 363699 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 363699 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 363699 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 363700 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 363700 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 363700 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 363701 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 363703 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 363704 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4]   2> 363704 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 363705 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 363706 T1371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d6d6ca name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 363706 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 363708 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 363711 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 363713 T1373 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ef514f name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 363713 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 363717 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 364720 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54846_
[junit4:junit4]   2> 364721 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54846_
[junit4:junit4]   2> 364723 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 364724 T1360 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 364724 T1373 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 364725 T1353 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 364728 T1374 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 364729 T1374 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 365032 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 365033 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:57702__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:57702_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:57702"}
[junit4:junit4]   2> 365037 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:54846_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:54846"}
[junit4:junit4]   2> 365037 T1354 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 365037 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 365041 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 365041 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 365041 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 365730 T1374 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1
[junit4:junit4]   2> 365730 T1374 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 365731 T1374 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 365731 T1374 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 365732 T1374 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1/'
[junit4:junit4]   2> 365733 T1374 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1/lib/README' to classloader
[junit4:junit4]   2> 365733 T1374 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 365772 T1374 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 365817 T1374 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 365819 T1374 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 365825 T1374 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 366196 T1374 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 366202 T1374 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 366205 T1374 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 366217 T1374 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 366220 T1374 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 366222 T1374 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 366223 T1374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 366223 T1374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 366224 T1374 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 366225 T1374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 366225 T1374 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 366225 T1374 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 366225 T1374 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1371889559010/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/
[junit4:junit4]   2> 366225 T1374 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4]   2> 366226 T1374 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 366226 T1374 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1
[junit4:junit4]   2> 366226 T1374 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index/
[junit4:junit4]   2> 366226 T1374 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 366227 T1374 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index
[junit4:junit4]   2> 366229 T1374 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 366229 T1374 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 366231 T1374 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 366231 T1374 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 366232 T1374 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 366232 T1374 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 366233 T1374 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 366233 T1374 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 366233 T1374 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 366233 T1374 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 366234 T1374 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 366236 T1374 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 366249 T1374 oass.SolrIndexSearcher.<init> Opening Searcher@1d9ec5a main
[junit4:junit4]   2> 366250 T1374 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 366250 T1374 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 366254 T1375 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d9ec5a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 366255 T1374 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 366255 T1374 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54846 collection:collection1 shard:shard1
[junit4:junit4]   2> 366256 T1374 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 366263 T1374 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 366266 T1374 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 366266 T1374 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 366266 T1374 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54846/collection1/
[junit4:junit4]   2> 366266 T1374 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 366266 T1374 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/ has no replicas
[junit4:junit4]   2> 366267 T1374 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54846/collection1/
[junit4:junit4]   2> 366267 T1374 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 366546 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 366553 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 366553 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 366553 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 366572 T1374 oasc.ZkController.register We are http://127.0.0.1:54846/collection1/ and leader is http://127.0.0.1:54846/collection1/
[junit4:junit4]   2> 366573 T1374 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54846
[junit4:junit4]   2> 366573 T1374 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 366573 T1374 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 366573 T1374 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 366576 T1374 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 366577 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 366577 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 366578 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 366641 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 366643 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35789
[junit4:junit4]   2> 366643 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 366644 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 366644 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024
[junit4:junit4]   2> 366644 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/solr.xml
[junit4:junit4]   2> 366644 T1331 oasc.CoreContainer.<init> New CoreContainer 17288485
[junit4:junit4]   2> 366645 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/'
[junit4:junit4]   2> 366645 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/'
[junit4:junit4]   2> 366690 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 366690 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 366691 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 366691 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 366691 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 366691 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 366692 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 366692 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 366692 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 366692 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 366695 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 366695 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4]   2> 366696 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 366696 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 366698 T1387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cf2cdb name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 366698 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 366700 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 366702 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 366704 T1389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1eeb9c6 name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 366704 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 366707 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 367710 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35789_
[junit4:junit4]   2> 367712 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35789_
[junit4:junit4]   2> 367714 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 367714 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 367715 T1360 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 367715 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 367715 T1389 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 367716 T1353 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 367716 T1373 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 367724 T1390 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 367724 T1390 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 368058 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 368059 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:54846__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:54846_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:54846"}
[junit4:junit4]   2> 368062 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:35789_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:35789"}
[junit4:junit4]   2> 368062 T1354 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 368062 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 368068 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 368068 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 368068 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 368071 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 368726 T1390 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1
[junit4:junit4]   2> 368726 T1390 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 368727 T1390 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 368727 T1390 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 368728 T1390 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1/'
[junit4:junit4]   2> 368729 T1390 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1/lib/README' to classloader
[junit4:junit4]   2> 368729 T1390 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 368762 T1390 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 368798 T1390 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 368800 T1390 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 368804 T1390 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 369155 T1390 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 369161 T1390 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 369163 T1390 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 369174 T1390 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 369177 T1390 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 369180 T1390 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 369181 T1390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 369181 T1390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 369181 T1390 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 369182 T1390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 369182 T1390 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 369182 T1390 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 369183 T1390 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1371889562024/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/
[junit4:junit4]   2> 369183 T1390 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4]   2> 369183 T1390 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 369183 T1390 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2
[junit4:junit4]   2> 369184 T1390 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index/
[junit4:junit4]   2> 369184 T1390 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 369184 T1390 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index
[junit4:junit4]   2> 369187 T1390 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 369187 T1390 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 369189 T1390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 369189 T1390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 369190 T1390 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 369190 T1390 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 369190 T1390 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 369191 T1390 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 369191 T1390 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 369191 T1390 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 369191 T1390 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 369194 T1390 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 369196 T1390 oass.SolrIndexSearcher.<init> Opening Searcher@169b6c0 main
[junit4:junit4]   2> 369197 T1390 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 369197 T1390 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 369200 T1391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@169b6c0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 369201 T1390 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 369202 T1390 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35789 collection:collection1 shard:shard1
[junit4:junit4]   2> 369205 T1390 oasc.ZkController.register We are http://127.0.0.1:35789/collection1/ and leader is http://127.0.0.1:54846/collection1/
[junit4:junit4]   2> 369205 T1390 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35789
[junit4:junit4]   2> 369206 T1390 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 369206 T1390 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C4795 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4795_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4]   2> 369206 T1392 C4795 P35789 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 369206 T1392 C4795 P35789 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 369207 T1390 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 369207 T1392 C4795 P35789 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 369207 T1392 C4795 P35789 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 369207 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 369208 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 369208 T1392 C4795 P35789 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 369208 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 369212 T1367 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:35789__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 369274 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 369276 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41066
[junit4:junit4]   2> 369277 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 369277 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 369277 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655
[junit4:junit4]   2> 369277 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/solr.xml
[junit4:junit4]   2> 369278 T1331 oasc.CoreContainer.<init> New CoreContainer 29850143
[junit4:junit4]   2> 369278 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/'
[junit4:junit4]   2> 369278 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/'
[junit4:junit4]   2> 369321 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 369321 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 369321 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 369322 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 369322 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 369322 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 369322 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 369323 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 369323 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 369323 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 369326 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 369326 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4]   2> 369326 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 369327 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 369329 T1404 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54c80c name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 369330 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 369332 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 369334 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 369335 T1406 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f53d3f name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 369336 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 369339 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 369575 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 369576 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:35789__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:35789_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:35789"}
[junit4:junit4]   2> 369579 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 369579 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 369579 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 369579 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 369580 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 370212 T1367 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:35789__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 370212 T1367 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:35789__collection1&state=recovering&nodeName=127.0.0.1:35789_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1000 
[junit4:junit4]   2> 370341 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41066_
[junit4:junit4]   2> 370342 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41066_
[junit4:junit4]   2> 370344 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 370345 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 370345 T1360 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 370344 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 370345 T1406 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 370345 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 370345 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 370346 T1353 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 370347 T1389 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 370347 T1373 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 370351 T1407 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 370351 T1407 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 371084 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 371085 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:41066_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:41066"}
[junit4:junit4]   2> 371085 T1354 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 371085 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 371089 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 371089 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 371089 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 371089 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 371089 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 371353 T1407 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1
[junit4:junit4]   2> 371353 T1407 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 371354 T1407 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 371354 T1407 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 371355 T1407 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1/'
[junit4:junit4]   2> 371355 T1407 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1/lib/README' to classloader
[junit4:junit4]   2> 371356 T1407 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 371386 T1407 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 371428 T1407 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 371429 T1407 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 371434 T1407 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 371798 T1407 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 371807 T1407 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 371810 T1407 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 371822 T1407 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 371826 T1407 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 371830 T1407 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 371831 T1407 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 371831 T1407 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 371831 T1407 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 371832 T1407 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 371832 T1407 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 371832 T1407 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 371832 T1407 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1371889564655/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/
[junit4:junit4]   2> 371833 T1407 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4]   2> 371833 T1407 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 371833 T1407 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3
[junit4:junit4]   2> 371833 T1407 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index/
[junit4:junit4]   2> 371834 T1407 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 371834 T1407 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index
[junit4:junit4]   2> 371836 T1407 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 371837 T1407 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 371838 T1407 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 371838 T1407 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 371839 T1407 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 371839 T1407 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 371840 T1407 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 371840 T1407 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 371840 T1407 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 371841 T1407 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 371841 T1407 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 371843 T1407 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 371845 T1407 oass.SolrIndexSearcher.<init> Opening Searcher@13819d3 main
[junit4:junit4]   2> 371846 T1407 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 371846 T1407 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 371850 T1408 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13819d3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 371852 T1407 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 371852 T1407 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41066 collection:collection1 shard:shard1
[junit4:junit4]   2> 371855 T1407 oasc.ZkController.register We are http://127.0.0.1:41066/collection1/ and leader is http://127.0.0.1:54846/collection1/
[junit4:junit4]   2> 371855 T1407 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41066
[junit4:junit4]   2> 371855 T1407 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 371855 T1407 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C4796 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4796_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4]   2> 371856 T1409 C4796 P41066 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 371856 T1409 C4796 P41066 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 371856 T1407 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 371856 T1409 C4796 P41066 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 371857 T1409 C4796 P41066 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 371857 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 371858 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 371858 T1409 C4796 P41066 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 371858 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 371862 T1366 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:41066__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 371921 T1331 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 371923 T1331 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37613
[junit4:junit4]   2> 371923 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 371923 T1331 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 371924 T1331 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304
[junit4:junit4]   2> 371924 T1331 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/solr.xml
[junit4:junit4]   2> 371924 T1331 oasc.CoreContainer.<init> New CoreContainer 12986159
[junit4:junit4]   2> 371924 T1331 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/'
[junit4:junit4]   2> 371925 T1331 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/'
[junit4:junit4]   2> 371970 T1331 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 371971 T1331 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 371971 T1331 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 371971 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 371971 T1331 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 371972 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 371972 T1331 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 371972 T1331 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 371972 T1331 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 371973 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 371975 T1331 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 371976 T1331 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42857/solr
[junit4:junit4]   2> 371976 T1331 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 371977 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 371978 T1421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46619f name:ZooKeeperConnection Watcher:127.0.0.1:42857 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 371979 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 371981 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 371983 T1331 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 371985 T1423 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@df0925 name:ZooKeeperConnection Watcher:127.0.0.1:42857/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 371985 T1331 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 371989 T1331 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C4797 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4797_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4]   2> 372213 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:54846/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 372213 T1392 C4797 P35789 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35789 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4]   2> 372214 T1392 C4797 P35789 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 372214 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 372214 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 372214 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 372214 T1392 C4797 P35789 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 372215 T1392 C4797 P35789 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:54846/collection1/. core=collection1
[junit4:junit4]   2> 372215 T1392 C4797 P35789 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4798 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4798_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4]   2> 372219 T1368 C4798 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 372221 T1367 C4798 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 372223 T1367 C4798 P54846 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 372223 T1367 C4798 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 372230 T1367 C4798 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 372230 T1367 C4798 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 372231 T1367 C4798 P54846 oass.SolrIndexSearcher.<init> Opening Searcher@1ece40a realtime
[junit4:junit4]   2> 372231 T1367 C4798 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 372231 T1367 C4798 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4]   2> 372232 T1392 C4797 P35789 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 372232 T1392 C4797 P35789 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 372233 T1368 C4798 P54846 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 372234 T1368 C4798 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 372234 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 372234 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 372234 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 372236 T1367 C4798 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 372237 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 372237 T1392 C4797 P35789 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681
[junit4:junit4]   2> 372238 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e49963 fullCopy=false
[junit4:junit4]   2> 372240 T1369 C4798 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 372241 T1392 C4797 P35789 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 372250 T1392 C4797 P35789 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 372250 T1392 C4797 P35789 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 372252 T1392 C4797 P35789 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 372252 T1392 C4797 P35789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 372252 T1392 C4797 P35789 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 372252 T1392 C4797 P35789 oass.SolrIndexSearcher.<init> Opening Searcher@115e129 main
[junit4:junit4]   2> 372253 T1391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@115e129 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 372254 T1392 C4797 P35789 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681;done=true>>]
[junit4:junit4]   2> 372254 T1392 C4797 P35789 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681
[junit4:junit4]   2> 372254 T1392 C4797 P35789 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index.20130622172607681
[junit4:junit4]   2> 372254 T1392 C4797 P35789 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 372254 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 372254 T1392 C4797 P35789 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 372255 T1392 C4797 P35789 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 372256 T1392 C4797 P35789 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 372594 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 372595 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:41066__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:41066_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:41066"}
[junit4:junit4]   2> 372599 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:35789__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:35789_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:35789"}
[junit4:junit4]   2> 372620 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 372620 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 372620 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 372620 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 372620 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 372620 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 372862 T1366 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:41066__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 372863 T1366 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:41066__collection1&state=recovering&nodeName=127.0.0.1:41066_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002 
[junit4:junit4]   2> 372992 T1331 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37613_
[junit4:junit4]   2> 372993 T1331 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37613_
[junit4:junit4]   2> 372995 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 372995 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 372995 T1360 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 372996 T1406 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 372996 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 372996 T1423 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 372997 T1353 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 372995 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 372997 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 372997 T1373 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 372996 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 372999 T1389 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 373004 T1426 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 373004 T1426 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 374131 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 374132 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37613_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37613"}
[junit4:junit4]   2> 374132 T1354 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 374132 T1354 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 374136 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 374136 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 374136 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 374136 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 374136 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 374136 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> ASYNC  NEW_CORE C4799 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4799_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4]   2> 374863 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:54846/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 374864 T1409 C4799 P41066 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41066 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4]   2> 374864 T1409 C4799 P41066 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 374864 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 374864 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 374865 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 374865 T1409 C4799 P41066 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 374865 T1409 C4799 P41066 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:54846/collection1/. core=collection1
[junit4:junit4]   2> 374865 T1409 C4799 P41066 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4800 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4800_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4]   2> 374867 T1369 C4800 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1 
[junit4:junit4]   2> 374869 T1368 C4800 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 374880 T1368 C4800 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 374880 T1368 C4800 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 374880 T1368 C4800 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 374881 T1368 C4800 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4]   2> 374881 T1409 C4799 P41066 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 374881 T1409 C4799 P41066 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 374884 T1366 C4800 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 374884 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4]   2> 374884 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 374884 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 374886 T1369 C4800 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0 
[junit4:junit4]   2> 374887 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 374887 T1409 C4799 P41066 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331
[junit4:junit4]   2> 374887 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331 lockFactory=org.apache.lucene.store.NativeFSLockFactory@19b2abb fullCopy=false
[junit4:junit4]   2> 374890 T1368 C4800 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0 
[junit4:junit4]   2> 374891 T1409 C4799 P41066 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 374893 T1409 C4799 P41066 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 374893 T1409 C4799 P41066 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 374894 T1409 C4799 P41066 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 374894 T1409 C4799 P41066 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 374895 T1409 C4799 P41066 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 374895 T1409 C4799 P41066 oass.SolrIndexSearcher.<init> Opening Searcher@19d6ea2 main
[junit4:junit4]   2> 374896 T1408 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19d6ea2 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 374896 T1409 C4799 P41066 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331;done=true>>]
[junit4:junit4]   2> 374896 T1409 C4799 P41066 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331
[junit4:junit4]   2> 374896 T1409 C4799 P41066 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index.20130622172610331
[junit4:junit4]   2> 374896 T1409 C4799 P41066 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 374897 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 374897 T1409 C4799 P41066 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 374897 T1409 C4799 P41066 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 374910 T1409 C4799 P41066 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 375007 T1426 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1
[junit4:junit4]   2> 375007 T1426 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 375008 T1426 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 375008 T1426 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 375009 T1426 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1/'
[junit4:junit4]   2> 375009 T1426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1/lib/README' to classloader
[junit4:junit4]   2> 375010 T1426 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 375041 T1426 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 375072 T1426 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 375074 T1426 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 375077 T1426 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 375461 T1426 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 375467 T1426 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 375470 T1426 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 375480 T1426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 375483 T1426 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 375489 T1426 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 375490 T1426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 375490 T1426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 375490 T1426 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 375491 T1426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 375491 T1426 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 375491 T1426 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 375492 T1426 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1371889567304/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/
[junit4:junit4]   2> 375492 T1426 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1bd9f3d
[junit4:junit4]   2> 375492 T1426 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 375493 T1426 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4
[junit4:junit4]   2> 375493 T1426 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index/
[junit4:junit4]   2> 375493 T1426 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 375494 T1426 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index
[junit4:junit4]   2> 375531 T1426 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 375532 T1426 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 375533 T1426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 375533 T1426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 375534 T1426 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 375534 T1426 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 375535 T1426 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 375535 T1426 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 375536 T1426 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 375536 T1426 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 375537 T1426 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 375539 T1426 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 375541 T1426 oass.SolrIndexSearcher.<init> Opening Searcher@1e410ff main
[junit4:junit4]   2> 375542 T1426 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 375542 T1426 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 375546 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e410ff main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 375548 T1426 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 375549 T1426 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37613 collection:collection1 shard:shard1
[junit4:junit4]   2> 375552 T1426 oasc.ZkController.register We are http://127.0.0.1:37613/collection1/ and leader is http://127.0.0.1:54846/collection1/
[junit4:junit4]   2> 375552 T1426 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37613
[junit4:junit4]   2> 375552 T1426 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 375553 T1426 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C4801 name=collection1 org.apache.solr.core.SolrCore@1a9d4fc url=http://127.0.0.1:37613/collection1 node=127.0.0.1:37613_ C4801_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4]   2> 375553 T1430 C4801 P37613 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 375553 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 375554 T1430 C4801 P37613 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 375554 T1426 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 375554 T1430 C4801 P37613 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 375555 T1331 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 375555 T1331 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 375555 T1430 C4801 P37613 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 375556 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 375559 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 375560 T1368 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:37613__collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 375560 T1331 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 375560 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 375561 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 375641 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 375642 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:41066__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:41066_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:41066"}
[junit4:junit4]   2> 375644 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:37613__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37613_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37613"}
[junit4:junit4]   2> 375647 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 375648 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 375648 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 375648 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 375647 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 375649 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 376560 T1368 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:37613__collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 376560 T1368 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=127.0.0.1:37613__collection1&state=recovering&nodeName=127.0.0.1:37613_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 376563 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 377564 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 377755 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=31836597
[junit4:junit4]   2>  C4801_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4]   2> 378561 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:54846/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 378561 T1430 C4801 P37613 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37613 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4]   2> 378562 T1430 C4801 P37613 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 378562 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 378562 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 378562 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 378562 T1430 C4801 P37613 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 378563 T1430 C4801 P37613 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:54846/collection1/. core=collection1
[junit4:junit4]   2> 378563 T1430 C4801 P37613 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4802 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4802_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4]   2> 378564 T1369 C4802 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 378566 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 378568 T1368 C4802 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 378581 T1368 C4802 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 378581 T1368 C4802 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 378582 T1368 C4802 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 378582 T1368 C4802 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4]   2> 378583 T1430 C4801 P37613 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 378584 T1430 C4801 P37613 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 378586 T1367 C4802 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 378586 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4]   2> 378587 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 378587 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 378589 T1366 C4802 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=0 
[junit4:junit4]   2> 378590 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 378591 T1430 C4801 P37613 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034
[junit4:junit4]   2> 378591 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c400a fullCopy=false
[junit4:junit4]   2> 378594 T1368 C4802 P54846 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0 
[junit4:junit4]   2> 378595 T1430 C4801 P37613 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 378597 T1430 C4801 P37613 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 378598 T1430 C4801 P37613 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 378600 T1430 C4801 P37613 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 378600 T1430 C4801 P37613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 378600 T1430 C4801 P37613 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 378601 T1430 C4801 P37613 oass.SolrIndexSearcher.<init> Opening Searcher@48157b main
[junit4:junit4]   2> 378602 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@48157b main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 378607 T1430 C4801 P37613 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034;done=true>>]
[junit4:junit4]   2> 378608 T1430 C4801 P37613 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034
[junit4:junit4]   2> 378608 T1430 C4801 P37613 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index.20130622172614034
[junit4:junit4]   2> 378608 T1430 C4801 P37613 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 378608 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 378608 T1430 C4801 P37613 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 378608 T1430 C4801 P37613 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 378611 T1430 C4801 P37613 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 378657 T1354 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 378658 T1354 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:37613__collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard_state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:37613_",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:37613"}
[junit4:junit4]   2> 378661 T1423 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 378662 T1353 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 378662 T1389 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 378662 T1373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 378662 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 378662 T1360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 379568 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 379569 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C4803 name=collection1 org.apache.solr.core.SolrCore@18c7b8d url=http://127.0.0.1:57702/collection1 node=127.0.0.1:57702_ C4803_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57702_, base_url=http://127.0.0.1:57702, leader=true}
[junit4:junit4]   2> 379575 T1348 C4803 P57702 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 379577 T1348 C4803 P57702 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 379577 T1348 C4803 P57702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 379650 T1348 C4803 P57702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 379650 T1348 C4803 P57702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 379651 T1348 C4803 P57702 oass.SolrIndexSearcher.<init> Opening Searcher@13423d0 main
[junit4:junit4]   2> 379651 T1348 C4803 P57702 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 379655 T1357 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13423d0 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> ASYNC  NEW_CORE C4804 name=collection1 org.apache.solr.core.SolrCore@18c7b8d url=http://127.0.0.1:57702/collection1 node=127.0.0.1:57702_ C4804_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57702_, base_url=http://127.0.0.1:57702, leader=true}
[junit4:junit4]   2> 379656 T1348 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 82
[junit4:junit4]   2> ASYNC  NEW_CORE C4805 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4805_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4]   2> 379666 T1382 C4805 P35789 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:54846/collection1/, StdNode: http://127.0.0.1:35789/collection1/, StdNode: http://127.0.0.1:41066/collection1/, StdNode: http://127.0.0.1:37613/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4806 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4806_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4]   2> 379669 T1368 C4806 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 379671 T1385 C4805 P35789 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C4807 name=collection1 org.apache.solr.core.SolrCore@1a9d4fc url=http://127.0.0.1:37613/collection1 node=127.0.0.1:37613_ C4807_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4]   2> 379673 T1419 C4807 P37613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C4808 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4808_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4]   2> 379676 T1399 C4808 P41066 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 379690 T1368 C4806 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=4
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 379690 T1368 C4806 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4]   2> 379691 T1368 C4806 P54846 oass.SolrIndexSearcher.<init> Opening Searcher@13ba920 main
[junit4:junit4]   2> 379691 T1368 C4806 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 379692 T1375 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13ba920 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 379692 T1368 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 23
[junit4:junit4]   2> 379720 T1419 C4807 P37613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 379720 T1399 C4808 P41066 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 379720 T1385 C4805 P35789 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 379721 T1399 C4808 P41066 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 379720 T1419 C4807 P37613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4]   2> 379721 T1385 C4805 P35789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 379722 T1399 C4808 P41066 oass.SolrIndexSearcher.<init> Opening Searcher@1d8bc2d main
[junit4:junit4]   2> 379723 T1419 C4807 P37613 oass.SolrIndexSearcher.<init> Opening Searcher@ed35dd main
[junit4:junit4]   2> 379723 T1385 C4805 P35789 oass.SolrIndexSearcher.<init> Opening Searcher@2a9f0e main
[junit4:junit4]   2> 379723 T1399 C4808 P41066 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 379723 T1408 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d8bc2d main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 379723 T1385 C4805 P35789 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 379723 T1419 C4807 P37613 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 379724 T1399 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 48
[junit4:junit4]   2> 379724 T1391 oasc.JmxMonitoredMap.put WARN Failed to register info bean: fieldValueCache javax.management.InstanceAlreadyExistsException: solr/collection1:type=fieldValueCache,id=org.apache.solr.search.FastLRUCache
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:303)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 
[junit4:junit4]   2> 379724 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ed35dd main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 379725 T1391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2a9f0e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 379726 T1419 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 53
[junit4:junit4]   2> 379726 T1385 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 55
[junit4:junit4]   2> 379727 T1382 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 61
[junit4:junit4]   2> 379728 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 379730 T1367 C4806 P54846 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 379731 T1384 C4805 P35789 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 379733 T1400 C4808 P41066 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 379734 T1417 C4807 P37613 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 381738 T1347 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1438530485282144256)} 0 1
[junit4:junit4]   2> 381744 T1382 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438530485286338560&update.from=http://127.0.0.1:54846/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438530485286338560)} 0 1
[junit4:junit4]   2> 381744 T1419 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438530485286338560&update.from=http://127.0.0.1:54846/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438530485286338560)} 0 1
[junit4:junit4]   2> 381746 T1402 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1438530485286338560&update.from=http://127.0.0.1:54846/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1438530485286338560)} 0 1
[junit4:junit4]   2> 381747 T1369 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1438530485286338560)} 0 6
[junit4:junit4]   2> 381747 T1399 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 8
[junit4:junit4]   2> 381751 T1348 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1438530485294727168)]} 0 2
[junit4:junit4]   2> 381761 T1417 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438530485301018624)]} 0 2
[junit4:junit4]   2> 381761 T1384 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438530485301018624)]} 0 2
[junit4:junit4]   2> 381763 T1401 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1438530485301018624)]} 0 1
[junit4:junit4]   2> 381764 T1367 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:41066/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1438530485301018624)]} 0 9
[junit4:junit4]   2> 381765 T1400 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 12
[junit4:junit4]   2> 381767 T1346 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1438530485313601536)]} 0 0
[junit4:junit4]   2> 381772 T1402 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438530485315698688)]} 0 0
[junit4:junit4]   2> 381773 T1383 C4805 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438530485315698688)]} 0 1
[junit4:junit4]   2> 381773 T1416 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1438530485315698688)]} 0 1
[junit4:junit4]   2> 381774 T1366 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1438530485315698688)]} 0 5
[junit4:junit4]   2> 381777 T1349 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1438530485323038720)]} 0 1
[junit4:junit4]   2> 381780 T1368 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:35789/collection1/]
[junit4:junit4]   2> 381781 T1368 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:35789/collection1/ against:[http://127.0.0.1:35789/collection1/] result:true
[junit4:junit4]   2> 381781 T1368 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:41066/collection1/ against:[http://127.0.0.1:35789/collection1/] result:false
[junit4:junit4]   2> 381781 T1368 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:37613/collection1/ against:[http://127.0.0.1:35789/collection1/] result:false
[junit4:junit4]   2> 381785 T1419 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[2 (1438530485328281600)]} 0 1
[junit4:junit4]   2> 381785 T1399 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[2 (1438530485328281600)]} 0 1
[junit4:junit4]   2> 381786 T1368 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:37613/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[2 (1438530485328281600)]} 0 6
[junit4:junit4]   2> 381786 T1418 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[2]} 0 7
[junit4:junit4]   2> 381790 T1347 C4804 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1438530485336670208)]} 0 1
[junit4:junit4]   2> 381795 T1369 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:35789/collection1/]
[junit4:junit4]   2> 381795 T1369 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:35789/collection1/ against:[http://127.0.0.1:35789/collection1/] result:true
[junit4:junit4]   2> 381796 T1369 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:41066/collection1/ against:[http://127.0.0.1:35789/collection1/] result:false
[junit4:junit4]   2> 381796 T1369 C4806 P54846 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:37613/collection1/ against:[http://127.0.0.1:35789/collection1/] result:false
[junit4:junit4]   2> 381800 T1417 C4807 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[3 (1438530485344010240)]} 0 1
[junit4:junit4]   2> 381800 T1400 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:54846/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[3 (1438530485344010240)]} 0 1
[junit4:junit4]   2> 381801 T1369 C4806 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:41066/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&wt=javabin&version=2} {add=[3 (1438530485344010240)]} 0 6
[junit4:junit4]   2> 381801 T1401 C4808 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:35789/collection1/&test.distrib.skip.servers=http://127.0.0.1:41066/collection1/&wt=javabin&version=2} {add=[3]} 0 8
[junit4:junit4]   2> 381803 T1348 C4804 P57702 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C4809 name=collection1 org.apache.solr.core.SolrCore@18c7b8d url=http://127.0.0.1:57702/collection1 node=127.0.0.1:57702_ C4809_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57702_, base_url=http://127.0.0.1:57702, leader=true}
[junit4:junit4]   2> 381830 T1348 C4809 P57702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 381830 T1348 C4809 P57702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 381834 T1348 C4809 P57702 oass.SolrIndexSearcher.<init> Opening Searcher@b39bb5 main
[junit4:junit4]   2> 381835 T1348 C4809 P57702 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 381835 T1357 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b39bb5 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 381836 T1348 C4809 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 33
[junit4:junit4]   2> ASYNC  NEW_CORE C4810 name=collection1 org.apache.solr.core.SolrCore@1a9d4fc url=http://127.0.0.1:37613/collection1 node=127.0.0.1:37613_ C4810_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4]   2> 381838 T1416 C4810 P37613 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:54846/collection1/, StdNode: http://127.0.0.1:35789/collection1/, StdNode: http://127.0.0.1:41066/collection1/, StdNode: http://127.0.0.1:37613/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4811 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4811_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4]   2> 381841 T1367 C4811 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C4812 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4812_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4]   2> 381841 T1384 C4812 P35789 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 381841 T1418 C4810 P37613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C4813 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4813_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4]   2> 381841 T1399 C4813 P41066 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 381900 T1399 C4813 P41066 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 381900 T1399 C4813 P41066 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 5
[junit4:junit4]   2> 381900 T1418 C4810 P37613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 381901 T1418 C4810 P37613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4]   2> 381900 T1367 C4811 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 381905 T1367 C4811 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4]   2> 381905 T1399 C4813 P41066 oass.SolrIndexSearcher.<init> Opening Searcher@499ec1 main
[junit4:junit4]   2> 381905 T1418 C4810 P37613 oass.SolrIndexSearcher.<init> Opening Searcher@1f6659e main
[junit4:junit4]   2> 381906 T1399 C4813 P41066 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 381906 T1429 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> 	at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> 	at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:299)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1846)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1723)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 
[junit4:junit4]   2> 381908 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f6659e main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 381906 T1418 C4810 P37613 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 381907 T1408 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@499ec1 main{StandardDirectoryReader(segments_5:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 381909 T1418 C4810 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 68
[junit4:junit4]   2> 381909 T1367 C4811 P54846 oass.SolrIndexSearcher.<init> Opening Searcher@1bf1de3 main
[junit4:junit4]   2> 381910 T1384 C4812 P35789 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty2/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 381910 T1384 C4812 P35789 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 381911 T1375 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1bf1de3 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 381910 T1399 C4813 P41066 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 69
[junit4:junit4]   2> 381910 T1367 C4811 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 381912 T1367 C4811 P54846 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 71
[junit4:junit4]   2> 381914 T1384 C4812 P35789 oass.SolrIndexSearcher.<init> Opening Searcher@10d4541 main
[junit4:junit4]   2> 381914 T1384 C4812 P35789 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 381915 T1391 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10d4541 main{StandardDirectoryReader(segments_4:3:nrt _0(5.0):C2)}
[junit4:junit4]   2> 381915 T1384 C4812 P35789 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 74
[junit4:junit4]   2> 381916 T1416 C4810 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 78
[junit4:junit4]   2> 381916 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 381917 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 381918 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 381920 T1366 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0 
[junit4:junit4]   2> 381922 T1383 C4812 P35789 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0 
[junit4:junit4]   2> 381925 T1400 C4813 P41066 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0 
[junit4:junit4]   2> 381927 T1417 C4810 P37613 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0 
[junit4:junit4]   2> 381927 T1331 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 381931 T1401 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4]   2> 381932 T1401 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 381936 T1368 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4]   2> 381936 T1368 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54846/collection1/
[junit4:junit4]   2> 381937 T1368 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54846 START replicas=[http://127.0.0.1:35789/collection1/, http://127.0.0.1:41066/collection1/, http://127.0.0.1:37613/collection1/] nUpdates=100
[junit4:junit4]   2> 381938 T1385 C4812 P35789 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 381939 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846  Received 3 versions from 127.0.0.1:35789/collection1/
[junit4:junit4]   2> 381939 T1419 C4810 P37613 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 381939 T1402 C4813 P41066 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 381939 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846  Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485315698688
[junit4:junit4]   2> 381940 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846  Received 5 versions from 127.0.0.1:37613/collection1/
[junit4:junit4]   2> 381940 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846  Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485328281600
[junit4:junit4]   2> 381940 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846  Received 5 versions from 127.0.0.1:41066/collection1/
[junit4:junit4]   2> 381941 T1368 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:54846  Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485328281600
[junit4:junit4]   2> 381941 T1368 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54846 DONE. sync succeeded
[junit4:junit4]   2> 381941 T1368 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 381941 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/: try and ask http://127.0.0.1:35789/collection1/ to sync
[junit4:junit4]   2> 381942 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/: try and ask http://127.0.0.1:41066/collection1/ to sync
[junit4:junit4]   2> 381942 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/: try and ask http://127.0.0.1:37613/collection1/ to sync
[junit4:junit4]   2> 381943 T1384 C4812 P35789 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35789 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4]   2> 381943 T1400 C4813 P41066 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41066 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4]   2> 381946 T1418 C4810 P37613 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37613 START replicas=[http://127.0.0.1:54846/collection1/] nUpdates=100
[junit4:junit4]   2> 381947 T1367 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 381947 T1366 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 381947 T1400 C4813 P41066 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:41066  Received 5 versions from 127.0.0.1:54846/collection1/
[junit4:junit4]   2> 381948 T1384 C4812 P35789 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:35789  Received 5 versions from 127.0.0.1:54846/collection1/
[junit4:junit4]   2> 381948 T1367 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 381948 T1384 C4812 P35789 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:35789 Requesting updates from 127.0.0.1:54846/collection1/n=2 versions=[1438530485344010240, 1438530485328281600]
[junit4:junit4]   2> 381948 T1400 C4813 P41066 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:41066  Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485328281600
[junit4:junit4]   2> 381949 T1400 C4813 P41066 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:41066 DONE. sync succeeded
[junit4:junit4]   2> 381949 T1418 C4810 P37613 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:37613  Received 5 versions from 127.0.0.1:54846/collection1/
[junit4:junit4]   2> 381949 T1400 C4813 P41066 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:54846/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6 
[junit4:junit4]   2> 381950 T1418 C4810 P37613 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:37613  Our versions are newer. ourLowThreshold=1438530485286338560 otherHigh=1438530485328281600
[junit4:junit4]   2> 381950 T1418 C4810 P37613 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37613 DONE. sync succeeded
[junit4:junit4]   2> 381950 T1418 C4810 P37613 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:54846/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=4 
[junit4:junit4]   2> 381951 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/:  sync completed with http://127.0.0.1:41066/collection1/
[junit4:junit4]   2> 381951 T1366 C4811 P54846 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1438530485344010240,1438530485328281600&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 381952 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/:  sync completed with http://127.0.0.1:37613/collection1/
[junit4:junit4]   2> 381954 T1384 C4812 P35789 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1438530485328281600), 3 (1438530485344010240)]} 0 2
[junit4:junit4]   2> 381954 T1384 C4812 P35789 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:35789 DONE. sync succeeded
[junit4:junit4]   2> 381955 T1384 C4812 P35789 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:54846/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=12 
[junit4:junit4]   2> 381955 T1368 oasc.SyncStrategy.syncToMe http://127.0.0.1:54846/collection1/:  sync completed with http://127.0.0.1:35789/collection1/
[junit4:junit4]   2> 381956 T1368 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=20 
[junit4:junit4]   2> 381956 T1401 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=25 
[junit4:junit4]   2> 381956 T1331 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 381957 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 381957 T1331 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 381958 T1331 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 381960 T1346 C4809 P57702 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 382010 T1346 C4809 P57702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/control/data/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 382010 T1346 C4809 P57702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 382011 T1346 C4809 P57702 oass.SolrIndexSearcher.<init> Opening Searcher@791cd4 main
[junit4:junit4]   2> 382011 T1346 C4809 P57702 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 382021 T1357 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@791cd4 main{StandardDirectoryReader(segments_3:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> ASYNC  NEW_CORE C4814 name=collection1 org.apache.solr.core.SolrCore@18c7b8d url=http://127.0.0.1:57702/collection1 node=127.0.0.1:57702_ C4814_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:57702_, base_url=http://127.0.0.1:57702, leader=true}
[junit4:junit4]   2> 382022 T1346 C4814 P57702 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 62
[junit4:junit4]   2> ASYNC  NEW_CORE C4815 name=collection1 org.apache.solr.core.SolrCore@6fd785 url=http://127.0.0.1:35789/collection1 node=127.0.0.1:35789_ C4815_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:35789_, base_url=http://127.0.0.1:35789}
[junit4:junit4]   2> 382024 T1383 C4815 P35789 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:54846/collection1/, StdNode: http://127.0.0.1:35789/collection1/, StdNode: http://127.0.0.1:41066/collection1/, StdNode: http://127.0.0.1:37613/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4816 name=collection1 org.apache.solr.core.SolrCore@43017e url=http://127.0.0.1:41066/collection1 node=127.0.0.1:41066_ C4816_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:41066_, base_url=http://127.0.0.1:41066}
[junit4:junit4]   2> 382027 T1402 C4816 P41066 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C4817 name=collection1 org.apache.solr.core.SolrCore@1a9d4fc url=http://127.0.0.1:37613/collection1 node=127.0.0.1:37613_ C4817_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:37613_, base_url=http://127.0.0.1:37613}
[junit4:junit4]   2> 382027 T1416 C4817 P37613 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> ASYNC  NEW_CORE C4818 name=collection1 org.apache.solr.core.SolrCore@1df5130 url=http://127.0.0.1:54846/collection1 node=127.0.0.1:54846_ C4818_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:54846_, base_url=http://127.0.0.1:54846, leader=true}
[junit4:junit4]   2> 382027 T1367 C4818 P54846 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 382028 T1385 C4815 P35789 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 382050 T1416 C4817 P37613 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty4/index,segFN=segments_7,generation=7}
[junit4:junit4]   2> 382050 T1416 C4817 P37613 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4]   2> 382051 T1416 C4817 P37613 oass.SolrIndexSearcher.<init> Opening Searcher@1b6784 main
[junit4:junit4]   2> 382051 T1416 C4817 P37613 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 382052 T1429 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b6784 main{StandardDirectoryReader(segments_6:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 382052 T1416 C4817 P37613 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 25
[junit4:junit4]   2> 382060 T1402 C4816 P41066 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_5,generation=5}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty3/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 382060 T1402 C4816 P41066 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4:junit4]   2> 382060 T1367 C4818 P54846 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=5
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_3,generation=3}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_4,generation=4}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_6,generation=6}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-1371889555429/jetty1/index,segFN=segments_7,generation=7}
[junit4:junit4]   2> 382061 T1367 C4818 P54846 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 7
[junit4:junit4]   2> 382062 T1367 C4818 P54846 oass.SolrIndexSearcher.<init> Opening Searcher@1e7f981 main
[junit4:junit4]   2> 382062 T1367 C4818 P54846 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2

[...truncated too long message...]

[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "collection":"collection1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:37613_",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:37613"}}}},
[junit4:junit4]   1>           "router":"compositeId"},
[junit4:junit4]   1>         "control_collection":{
[junit4:junit4]   1>           "shards":{"shard1":{
[junit4:junit4]   1>               "range":"80000000-7fffffff",
[junit4:junit4]   1>               "state":"active",
[junit4:junit4]   1>               "replicas":{"127.0.0.1:57702__collection1":{
[junit4:junit4]   1>                   "shard":"shard1",
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "collection":"control_collection",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:57702_",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:57702",
[junit4:junit4]   1>                   "leader":"true"}}}},
[junit4:junit4]   1>           "router":"compositeId"}}
[junit4:junit4]   1>   /solr/aliases.json (0)
[junit4:junit4]   1>   /solr/live_nodes (4)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:54846_ (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:41066_ (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:57702_ (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:37613_ (0)
[junit4:junit4]   1>   /solr/overseer (3)
[junit4:junit4]   1>   DATA:
[junit4:junit4]   1>       
[junit4:junit4]   1>    /solr/overseer/queue (0)
[junit4:junit4]   1>    /solr/overseer/queue-work (0)
[junit4:junit4]   1>    /solr/overseer/collection-queue-work (0)
[junit4:junit4]   1>   /solr/collections (2)
[junit4:junit4]   1>    /solr/collections/collection1 (3)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"configName":"conf1"}
[junit4:junit4]   1>     /solr/collections/collection1/shards (0)
[junit4:junit4]   1>     /solr/collections/collection1/leader_elect (1)
[junit4:junit4]   1>      /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4]   1>       /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89908153904726030-127.0.0.1:54846__collection1-n_0000000004 (0)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89908153904726026-127.0.0.1:41066__collection1-n_0000000002 (0)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89908153904726028-127.0.0.1:37613__collection1-n_0000000003 (0)
[junit4:junit4]   1>     /solr/collections/collection1/leaders (1)
[junit4:junit4]   1>      /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4]   1>      DATA:
[junit4:junit4]   1>          {
[junit4:junit4]   1>            "core":"collection1",
[junit4:junit4]   1>            "node_name":"127.0.0.1:41066_",
[junit4:junit4]   1>            "base_url":"http://127.0.0.1:41066"}
[junit4:junit4]   1>    /solr/collections/control_collection (3)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"configName":"conf1"}
[junit4:junit4]   1>     /solr/collections/control_collection/shards (0)
[junit4:junit4]   1>     /solr/collections/control_collection/leader_elect (1)
[junit4:junit4]   1>      /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4]   1>       /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4]   1>        /solr/collections/control_collection/leader_elect/shard1/election/89908153904726019-127.0.0.1:57702__collection1-n_0000000000 (0)
[junit4:junit4]   1>     /solr/collections/control_collection/leaders (1)
[junit4:junit4]   1>      /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4]   1>      DATA:
[junit4:junit4]   1>          {
[junit4:junit4]   1>            "core":"collection1",
[junit4:junit4]   1>            "node_name":"127.0.0.1:57702_",
[junit4:junit4]   1>            "base_url":"http://127.0.0.1:57702"}
[junit4:junit4]   1>   /solr/overseer_elect (2)
[junit4:junit4]   1>    /solr/overseer_elect/election (4)
[junit4:junit4]   1>     /solr/overseer_elect/election/89908153904726028-127.0.0.1:37613_-n_0000000004 (0)
[junit4:junit4]   1>     /solr/overseer_elect/election/89908153904726019-127.0.0.1:57702_-n_0000000000 (0)
[junit4:junit4]   1>     /solr/overseer_elect/election/89908153904726030-127.0.0.1:54846_-n_0000000005 (0)
[junit4:junit4]   1>     /solr/overseer_elect/election/89908153904726026-127.0.0.1:41066_-n_0000000003 (0)
[junit4:junit4]   1>    /solr/overseer_elect/leader (0)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"id":"89908153904726019-127.0.0.1:57702_-n_0000000000"}
[junit4:junit4]   1>  /zookeeper (1)
[junit4:junit4]   1>  DATA:
[junit4:junit4]   1>      
[junit4:junit4]   1> 
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=CB968363466C6444 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PA -Dtests.timezone=Asia/Tokyo -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE  101s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:54846/collection1lastClient and got 247 from http://127.0.0.1:41066/collection1
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([CB968363466C6444:4A700D7B31330478]:0)
[junit4:junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1018)
[junit4:junit4]    > 	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:238)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 461253 T1331 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 101277 T1330 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4]   2> 461310 T1406 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 461311 T1406 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 461311 T1406 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {range_facet_l=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), text=PostingsFormat(name=Direct), _version_=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), rnd_b=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), intDefault=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), timestamp=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), id=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), a_t=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), range_facet_sl=PostingsFormat(name=Direct), range_facet_si=PostingsFormat(name=SimpleText), other_tl1=Pulsing41(freqCutoff=6 minBlockSize=66 maxBlockSize=163), multiDefault=PostingsFormat(name=SimpleText), a_si=PostingsFormat(name=SimpleText)}, docValues:{timestamp=DocValuesFormat(name=CheapBastard)}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=es_PA, timezone=Asia/Tokyo
[junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.7.0_25 (32-bit)/cpus=8,threads=1,free=25810176,total=100057088
[junit4:junit4]   2> NOTE: All tests run in this JVM: [DocValuesTest, SpellingQueryConverterTest, TestPerFieldSimilarity, TestPropInjectDefaults, SpatialFilterTest, SynonymTokenizerTest, AlternateDirectoryTest, TestPseudoReturnFields, TestGroupingSearch, TestStressVersions, TestReversedWildcardFilterFactory, ClusterStateUpdateTest, PreAnalyzedFieldTest, TestSolrQueryParser, MBeansHandlerTest, TestSolrIndexConfig, TermsComponentTest, TestDocSet, SuggesterFSTTest, JsonLoaderTest, RecoveryZkTest, DirectSolrConnectionTest, SOLR749Test, TestStressRecovery, ResourceLoaderTest, RegexBoostProcessorTest, FieldMutatingUpdateProcessorTest, TestPhraseSuggestions, TestReplicationHandler, TestPluginEnable, DistributedQueryElevationComponentTest, TestJoin, TestBadConfig, TestUniqueKeyFieldResource, AliasIntegrationTest, TestFastWriter, SliceStateUpdateTest, TestLRUCache, QueryEqualityTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 101.80s, 1 test, 1 failure <<< FAILURES!

[...truncated 620 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 297 suites, 1253 tests, 1 failure, 13 ignored (7 assumptions)

Total time: 40 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_25 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message