lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Jenkins Server <jenk...@builds.apache.org>
Subject [JENKINS] Lucene-Solr-tests-only-trunk-java7 - Build # 2965 - Still Failing
Date Wed, 25 Jul 2012 09:51:06 GMT
Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/2965/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.RecoveryZkTest.testDistribSearch

Error Message:
There are still nodes recoverying

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying
	at __randomizedtesting.SeedInfo.seed([3D6B3CDC778FA2A:82303DD5B0279A16]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistributedZkTestCase.waitForRecoveriesToFinish(AbstractDistributedZkTestCase.java:153)
	at org.apache.solr.cloud.AbstractDistributedZkTestCase.waitForRecoveriesToFinish(AbstractDistributedZkTestCase.java:115)
	at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:87)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:679)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:818)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:825)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(RandomizedRunner.java:132)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(RandomizedRunner.java:671)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:697)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:736)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:747)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
	at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ERROR: SolrIndexSearcher opens=76 closes=75

Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=76 closes=75
	at __randomizedtesting.SeedInfo.seed([3D6B3CDC778FA2A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:216)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:82)
	at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:754)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
	at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551)




Build Log:
[...truncated 10034 lines...]
[junit4:junit4]   2> 210 T1129 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 210 T1129 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 210 T1129 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 211 T1129 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 221 T1129 oash.ReplicationHandler.inform Replication enabled for following config files: schema.xml
[junit4:junit4]   2> 221 T1129 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 221 T1129 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 221 T1139 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@76d319d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 221 T1129 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test/J1
[junit4:junit4]   2> 222 T1129 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 222 T1129 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 235 T1129 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 237 T1129 oejs.AbstractConnector.doStart Started SocketConnector@0.0.0.0:42876
[junit4:junit4]   2> 238 T1129 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 238 T1129 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave
[junit4:junit4]   2> 238 T1129 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/'
[junit4:junit4]   2> 244 T1129 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 245 T1129 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 245 T1129 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave
[junit4:junit4]   2> 245 T1129 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/solr.xml
[junit4:junit4]   2> 245 T1129 oasc.CoreContainer.<init> New CoreContainer 237220211
[junit4:junit4]   2> 245 T1129 oasc.CoreContainer$Initializer.initialize no solr.xml file found - using default
[junit4:junit4]   2> 246 T1129 oasc.CoreContainer.load Loading CoreContainer using Solr Home: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/'
[junit4:junit4]   2> 246 T1129 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/'
[junit4:junit4]   2> 256 T1129 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 277 T1129 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/collection1
[junit4:junit4]   2> 277 T1129 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/collection1/'
[junit4:junit4]   2> 308 T1129 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 382 T1129 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 383 T1129 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 386 T1129 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 407 T1129 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 409 T1129 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/collection1/, dataDir=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/collection1/data/
[junit4:junit4]   2> 409 T1129 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 409 T1129 oasc.SolrCore.getNewIndexDir WARNING New index directory detected: old=null new=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/collection1/data/index/
[junit4:junit4]   2> 410 T1129 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/collection1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 413 T1129 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test/J1/org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/collection1/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 414 T1129 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 415 T1129 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 415 T1129 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 415 T1129 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 415 T1129 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 416 T1129 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 416 T1129 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 417 T1129 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 420 T1129 oass.SolrIndexSearcher.<init> Opening Searcher@45d39891 main
[junit4:junit4]   2> 421 T1129 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 421 T1129 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 421 T1129 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 422 T1129 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 422 T1129 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 422 T1129 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 422 T1129 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 423 T1129 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 423 T1129 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 423 T1129 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 423 T1129 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 424 T1129 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 435 T1129 oash.SnapPuller.startExecutorService Poll Scheduled at an interval of 1000ms
[junit4:junit4]   2> 435 T1149 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@45d39891 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 435 T1129 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 435 T1129 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test/J1
[junit4:junit4]   2> 436 T1129 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 436 T1129 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 
[junit4:junit4]    > (@AfterClass output)
[junit4:junit4]   2> 32468 T1129 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=37538724
[junit4:junit4]   2> 32468 T1129 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7e8904
[junit4:junit4]   2> 32468 T1129 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 32469 T1129 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=493,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 32470 T1129 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 32522 T1129 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=544619397
[junit4:junit4]   2> 32522 T1129 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5d3f4031
[junit4:junit4]   2> 32522 T1129 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 32523 T1129 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 32524 T1129 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 32585 T1129 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 34723 T1331 oash.SnapPuller.fetchLatestIndex SEVERE Master at: http://localhost:62491/solr is not available. Index fetch failed. Exception: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://localhost:62491/solr
[junit4:junit4]   2> 34724 T1331 oash.SnapPuller.logReplicationTimeAndConfFiles WARNING Exception while updating statistics java.io.FileNotFoundException: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1343208842511/slave/collection1/data/replication.properties (No such file or directory)
[junit4:junit4]   2> 	at java.io.FileOutputStream.open(Native Method)
[junit4:junit4]   2> 	at java.io.FileOutputStream.<init>(FileOutputStream.java:212)
[junit4:junit4]   2> 	at java.io.FileOutputStream.<init>(FileOutputStream.java:165)
[junit4:junit4]   2> 	at org.apache.solr.handler.SnapPuller.logReplicationTimeAndConfFiles(SnapPuller.java:480)
[junit4:junit4]   2> 	at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:406)
[junit4:junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:297)
[junit4:junit4]   2> 	at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:175)
[junit4:junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
[junit4:junit4]   2> 	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
[junit4:junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
[junit4:junit4]   2> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
[junit4:junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 
[junit4:junit4]   2> 39418 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 40808 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 51520 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 53453 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 63622 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 65063 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 75724 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 77370 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 87826 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 89047 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 99928 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 101077 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 112030 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 113850 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 124132 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 125475 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 136234 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 137893 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 148336 T958 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7a6f2a0010, closing socket connection and attempting reconnect
[junit4:junit4]   2> 149971 T958 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:49429
[junit4:junit4]   2> 153706 T1129 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE ERROR: SolrIndexSearcher opens=76 closes=75
[junit4:junit4]   2> NOTE: test params are: codec=Lucene40: {newname=MockVariableIntBlock(baseBlockSize=27), id=MockVariableIntBlock(baseBlockSize=27), name=MockVariableIntBlock(baseBlockSize=27)}, sim=DefaultSimilarity, locale=es_US, timezone=Pacific/Tongatapu
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Oracle Corporation 1.7.0_04 (64-bit)/cpus=16,threads=1,free=291723920,total=452984832
[junit4:junit4]   2> NOTE: All tests run in this JVM: [IndexSchemaRuntimeFieldTest, FileBasedSpellCheckerTest, TestPropInjectDefaults, TestSystemIdResolver, SortByFunctionTest, TestSuggestSpellingConverter, TestDocumentBuilder, TestShingleFilterFactory, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, MBeansHandlerTest, UUIDFieldTest, TestQuerySenderNoQuery, TestRussianLightStemFilterFactory, LeaderElectionTest, TestRealTimeGet, TestLRUCache, TestSolrDeletionPolicy2, TestGermanMinimalStemFilterFactory, TestHungarianLightStemFilterFactory, StandardRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TestDocSet, SampleTest, TestMultiCoreConfBootstrap, TestNorwegianMinimalStemFilterFactory, TestCSVResponseWriter, FullSolrCloudTest, DistributedTermsComponentTest, TestXIncludeConfig, ZkControllerTest, JsonLoaderTest, SpellingQueryConverterTest, TestGreekStemFilterFactory, PolyFieldTest, IndexReaderFactoryTest, TestUAX29URLEmailTokenizerFactory, TestReplicationHandler]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicationHandler -Dtests.seed=3D6B3CDC778FA2A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_US -Dtests.timezone=Pacific/Tongatapu -Dtests.file.encoding=US-ASCII
[junit4:junit4]   2> 
[junit4:junit4] ERROR   0.00s J1 | TestReplicationHandler (suite)
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=76 closes=75
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([3D6B3CDC778FA2A]:0)
[junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:216)
[junit4:junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:82)
[junit4:junit4]    > 	at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:601)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:754)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551)
[junit4:junit4]    >
[junit4:junit4] Completed on J1 in 153.75s, 1 test, 1 failure <<< FAILURES!
[...truncated 145263 lines...]
[junit4:junit4]   2> 405353 T1802 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=45,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=25426,adds=234,deletesById=120,deletesByQuery=0,errors=43,cumulative_adds=29055,cumulative_deletesById=14382,cumulative_deletesByQuery=0,cumulative_errors=1}
[junit4:junit4]   2> 405353 T1802 C97 P40947 oasu.DirectUpdateHandler2.closeWriter SEVERE Error in final commit java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit
[junit4:junit4]   2> 	at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2541)
[junit4:junit4]   2> 	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2699)
[junit4:junit4]   2> 	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2679)
[junit4:junit4]   2> 	at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:715)
[junit4:junit4]   2> 	at org.apache.solr.update.DefaultSolrCoreState.decref(DefaultSolrCoreState.java:130)
[junit4:junit4]   2> 	at org.apache.solr.update.DirectUpdateHandler2.close(DirectUpdateHandler2.java:662)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.close(SolrCore.java:864)
[junit4:junit4]   2> 	at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:590)
[junit4:junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:129)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:162)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:432)
[junit4:junit4]   2> 	at org.apache.solr.cloud.FullSolrCloudTest.destroyServers(FullSolrCloudTest.java:1466)
[junit4:junit4]   2> 	at org.apache.solr.BaseDistributedSearchTestCase.tearDown(BaseDistributedSearchTestCase.java:182)
[junit4:junit4]   2> 	at org.apache.solr.cloud.AbstractDistributedZkTestCase.tearDown(AbstractDistributedZkTestCase.java:204)
[junit4:junit4]   2> 	at org.apache.solr.cloud.FullSolrCloudTest.tearDown(FullSolrCloudTest.java:1451)
[junit4:junit4]   2> 	at org.apache.solr.cloud.RecoveryZkTest.tearDown(RecoveryZkTest.java:134)
[junit4:junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:601)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:899)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32)
[junit4:junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:825)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(RandomizedRunner.java:132)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(RandomizedRunner.java:671)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:697)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:736)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:747)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551)
[junit4:junit4]   2> 
[junit4:junit4]   2> 406513 T1835 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 13321ms for sessionid 0x138bd7f08410003, closing socket connection and attempting reconnect
[junit4:junit4]   2> 406747 T3408 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:40941
[junit4:junit4]   2> 407729 T1835 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:40941
[junit4:junit4]   2> 417295 T3408 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7f08410007, closing socket connection and attempting reconnect
[junit4:junit4]   2> 418716 T1836 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 418716 T1802 oaz.ZooKeeper.close Session: 0x138bd7f08410003 closed
[junit4:junit4]   2> 418716 T1802 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 418769 T1802 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 40949
[junit4:junit4]   2> 418769 T1802 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=51802876
[junit4:junit4]   2> 418769 T1802 oasc.RecoveryStrategy.close WARNING Stopping recovery for core collection1 zkNodeName=127.0.0.1:40949_solr_collection1
[junit4:junit4]   2> 418811 T3408 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:40941
[junit4:junit4]   2> 429397 T3408 oaz.ClientCnxn$SendThread.run Client session timed out, have not heard from server in 12001ms for sessionid 0x138bd7f08410007, closing socket connection and attempting reconnect
[junit4:junit4]   2> 429498 T7985 C98 P40949 oasc.RecoveryStrategy.doRecovery SEVERE Error while trying to recover. java.lang.RuntimeException: Interrupted
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:78)
[junit4:junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:241)
[junit4:junit4]   2> 	at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:236)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ZkController.publish(ZkController.java:734)
[junit4:junit4]   2> 	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:288)
[junit4:junit4]   2> 	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:210)
[junit4:junit4]   2> 
[junit4:junit4]   2> 429498 T7985 C98 P40949 oasc.RecoveryStrategy.doRecovery SEVERE Recovery failed - trying again...
[junit4:junit4]   2> 429498 T7985 C98 P40949 oasc.RecoveryStrategy.doRecovery Finished recovery process
[junit4:junit4]   2> 429498 T1802 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@59a10ff1
[junit4:junit4]   2> 429501 T1802 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 429502 T1802 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=25230,adds=25231,deletesById=12508,deletesByQuery=0,errors=2,cumulative_adds=27050,cumulative_deletesById=13420,cumulative_deletesByQuery=0,cumulative_errors=2}
[junit4:junit4]   2> 429502 T1802 C98 P40949 oasu.DirectUpdateHandler2.closeWriter SEVERE Error in final commit java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit
[junit4:junit4]   2> 	at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2541)
[junit4:junit4]   2> 	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2699)
[junit4:junit4]   2> 	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2679)
[junit4:junit4]   2> 	at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:715)
[junit4:junit4]   2> 	at org.apache.solr.update.DefaultSolrCoreState.decref(DefaultSolrCoreState.java:130)
[junit4:junit4]   2> 	at org.apache.solr.update.DirectUpdateHandler2.close(DirectUpdateHandler2.java:662)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.close(SolrCore.java:864)
[junit4:junit4]   2> 	at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:590)
[junit4:junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:129)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:162)
[junit4:junit4]   2> 	at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:432)
[junit4:junit4]   2> 	at org.apache.solr.cloud.FullSolrCloudTest.destroyServers(FullSolrCloudTest.java:1466)
[junit4:junit4]   2> 	at org.apache.solr.BaseDistributedSearchTestCase.tearDown(BaseDistributedSearchTestCase.java:182)
[junit4:junit4]   2> 	at org.apache.solr.cloud.AbstractDistributedZkTestCase.tearDown(AbstractDistributedZkTestCase.java:204)
[junit4:junit4]   2> 	at org.apache.solr.cloud.FullSolrCloudTest.tearDown(FullSolrCloudTest.java:1451)
[junit4:junit4]   2> 	at org.apache.solr.cloud.RecoveryZkTest.tearDown(RecoveryZkTest.java:134)
[junit4:junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:601)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:899)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32)
[junit4:junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:825)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(RandomizedRunner.java:132)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(RandomizedRunner.java:671)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:697)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:736)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:747)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132)
[junit4:junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551)
[junit4:junit4]   2> 
[junit4:junit4]   2> 431206 T3408 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:40941
[junit4:junit4]   2> 432125 T1802 oasc.RecoveryStrategy.close WARNING Stopping recovery for core collection1 zkNodeName=127.0.0.1:40949_solr_collection1
[junit4:junit4]   2> 441600 T3409 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 441600 T1802 oaz.ZooKeeper.close Session: 0x138bd7f08410007 closed
[junit4:junit4]   2> 441601 T1802 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 441789 T1802 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RecoveryZkTest -Dtests.method=testDistribSearch -Dtests.seed=3D6B3CDC778FA2A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_EC -Dtests.timezone=EET -Dtests.file.encoding=US-ASCII
[junit4:junit4]   2>
[junit4:junit4]    > (@AfterClass output)
[junit4:junit4]   2> 441832 T1802 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> NOTE: test params are: codec=Lucene40: {range_facet_l=PostingsFormat(name=Asserting), text=PostingsFormat(name=Lucene40WithOrds), _version_=MockFixedIntBlock(blockSize=836), rnd_b=PostingsFormat(name=Direct), intDefault=MockFixedIntBlock(blockSize=836), id=PostingsFormat(name=Direct), timestamp=PostingsFormat(name=Asserting), a_t=PostingsFormat(name=Asserting), range_facet_sl=PostingsFormat(name=Direct), range_facet_si=PostingsFormat(name=Asserting), other_tl1=MockFixedIntBlock(blockSize=836), multiDefault=PostingsFormat(name=Asserting), a_si=MockFixedIntBlock(blockSize=836)}, sim=RandomSimilarityProvider(queryNorm=false,coord=false): {}, locale=es_EC, timezone=EET
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Oracle Corporation 1.7.0_04 (64-bit)/cpus=16,threads=2,free=130501360,total=536870912
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestJapaneseBaseFormFilterFactory, DateFieldTest, TestPortugueseStemFilterFactory, BinaryUpdateRequestHandlerTest, LegacyHTMLStripCharFilterTest, FileUtilsTest, CommonGramsQueryFilterFactoryTest, TestPerFieldSimilarity, TestGermanStemFilterFactory, TestHashPartitioner, EchoParamsTest, BasicFunctionalityTest, SuggesterTSTTest, CacheHeaderTest, CoreAdminHandlerTest, TestFoldingMultitermQuery, TestBM25SimilarityFactory, TestSearchPerf, TestJapaneseKatakanaStemFilterFactory, TestBadConfig, StatelessScriptUpdateProcessorFactoryTest, TestBulgarianStemFilterFactory, TestJapanesePartOfSpeechStopFilterFactory, TestPhoneticFilterFactory, ShowFileRequestHandlerTest, XmlUpdateRequestHandlerTest, IndexBasedSpellCheckerTest, TestWordDelimiterFilterFactory, TestTrimFilterFactory, AutoCommitTest, TestPatternReplaceCharFilterFactory, PluginInfoTest, PreAnalyzedFieldTest, SyncSliceTest, TestPortugueseMinimalStemFilterFactory, TermsComponentTest, TestPropInject, JSONWriterTest, LukeRequestHandlerTest, DateMathParserTest, OutputWriterTest, MoreLikeThisHandlerTest, TestHTMLStripCharFilterFactory, DefaultValueUpdateProcessorTest, DirectUpdateHandlerOptimizeTest, TestCapitalizationFilterFactory, TestFiltering, TestPhraseSuggestions, DisMaxRequestHandlerTest, TestLFUCache, TestRemoveDuplicatesTokenFilterFactory, TestStemmerOverrideFilterFactory, SolrRequestParserTest, TestCharFilters, MinimalSchemaTest, PingRequestHandlerTest, CopyFieldTest, TestQueryUtils, PrimitiveFieldTypeTest, TestPHPSerializedResponseWriter, TestJmxIntegration, FastVectorHighlighterTest, TestGalicianStemFilterFactory, HighlighterTest, BadComponentTest, AnalysisAfterCoreReloadTest, DirectSolrSpellCheckerTest, TestArabicFilters, TestSwedishLightStemFilterFactory, TestQuerySenderListener, TestReversedWildcardFilterFactory, DoubleMetaphoneFilterFactoryTest, BasicDistributedZkTest, OverseerTest, RecoveryZkTest]
[junit4:junit4]   2> 
[junit4:junit4] Completed on J0 in 441.84s, 1 test, 1 failure <<< FAILURES!
[...truncated 435 lines...]

[...truncated 156034 lines...]

[...truncated 156034 lines...]

[...truncated 156034 lines...]

[...truncated 156034 lines...]

[...truncated 10150 lines...]
[junit4:junit4]   2> 153706 T1129 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE ERROR: SolrIndexSearcher opens=76 closes=75
[junit4:junit4]   2> NOTE: test params are: codec=Lucene40: {newname=MockVariableIntBlock(baseBlockSize=27), id=MockVariableIntBlock(baseBlockSize=27), name=MockVariableIntBlock(baseBlockSize=27)}, sim=DefaultSimilarity, locale=es_US, timezone=Pacific/Tongatapu
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Oracle Corporation 1.7.0_04 (64-bit)/cpus=16,threads=1,free=291723920,total=452984832
[junit4:junit4]   2> NOTE: All tests run in this JVM: [IndexSchemaRuntimeFieldTest, FileBasedSpellCheckerTest, TestPropInjectDefaults, TestSystemIdResolver, SortByFunctionTest, TestSuggestSpellingConverter, TestDocumentBuilder, TestShingleFilterFactory, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, MBeansHandlerTest, UUIDFieldTest, TestQuerySenderNoQuery, TestRussianLightStemFilterFactory, LeaderElectionTest, TestRealTimeGet, TestLRUCache, TestSolrDeletionPolicy2, TestGermanMinimalStemFilterFactory, TestHungarianLightStemFilterFactory, StandardRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TestDocSet, SampleTest, TestMultiCoreConfBootstrap, TestNorwegianMinimalStemFilterFactory, TestCSVResponseWriter, FullSolrCloudTest, DistributedTermsComponentTest, TestXIncludeConfig, ZkControllerTest, JsonLoaderTest, SpellingQueryConverterTest, TestGreekStemFilterFactory, PolyFieldTest, IndexReaderFactoryTest, TestUAX29URLEmailTokenizerFactory, TestReplicationHandler]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicationHandler -Dtests.seed=3D6B3CDC778FA2A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_US -Dtests.timezone=Pacific/Tongatapu -Dtests.file.encoding=US-ASCII
[junit4:junit4]   2> 
[junit4:junit4] ERROR   0.00s J1 | TestReplicationHandler (suite)
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=76 closes=75
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([3D6B3CDC778FA2A]:0)
[junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:216)
[junit4:junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:82)
[junit4:junit4]    > 	at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:601)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:754)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551)
[junit4:junit4]    >
[junit4:junit4] Completed on J1 in 153.75s, 1 test, 1 failure <<< FAILURES!
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4:junit4] Completed on J0 in 37.51s, 8 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.search.TestStressReorder
[junit4:junit4] Completed on J1 in 11.43s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestBeiderMorseFilterFactory
[junit4:junit4] Completed on J1 in 0.57s, 3 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.search.TestIndexSearcher
[junit4:junit4] Completed on J1 in 2.12s, 2 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestStopFilterFactory
[junit4:junit4] Completed on J1 in 0.03s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestNGramFilters
[junit4:junit4] Completed on J1 in 0.08s, 10 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestKeywordMarkerFilterFactory
[junit4:junit4] Completed on J1 in 0.02s, 2 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.core.TestConfig
[junit4:junit4] Completed on J1 in 0.34s, 7 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.schema.IndexSchemaTest
[junit4:junit4] Completed on J1 in 1.26s, 3 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.schema.MultiTermTest
[junit4:junit4] Completed on J1 in 0.76s, 3 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.search.TestRangeQuery
[junit4:junit4] Completed on J1 in 6.87s, 2 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestPersianNormalizationFilterFactory
[junit4:junit4] Completed on J1 in 0.02s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestReverseStringFilterFactory
[junit4:junit4] Completed on J1 in 0.03s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.core.TestSolrDeletionPolicy1
[junit4:junit4] Completed on J1 in 1.35s, 3 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.internal.csv.CSVParserTest
[junit4:junit4] Completed on J1 in 0.10s, 23 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestElisionFilterFactory
[junit4:junit4] Completed on J1 in 0.03s, 3 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestTurkishLowerCaseFilterFactory
[junit4:junit4] Completed on J1 in 0.01s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.schema.TestCollationField
[junit4:junit4] Completed on J1 in 0.52s, 8 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.search.QueryParsingTest
[junit4:junit4] Completed on J1 in 1.11s, 4 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.TestPluginEnable
[junit4:junit4] Completed on J1 in 0.16s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.schema.TestOmitPositions
[junit4:junit4] Completed on J1 in 0.85s, 2 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.search.TestPseudoReturnFields
[junit4:junit4] Completed on J1 in 1.62s, 13 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] Completed on J1 in 0.00s, 0 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.update.processor.UniqFieldsUpdateProcessorFactoryTest
[junit4:junit4] Completed on J1 in 0.74s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.search.TestRecovery
[junit4:junit4] Completed on J1 in 11.08s, 9 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.spelling.WordBreakSolrSpellCheckerTest
[junit4:junit4] Completed on J1 in 1.65s, 3 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestCJKBigramFilterFactory
[junit4:junit4] Completed on J1 in 0.02s, 2 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.handler.CSVRequestHandlerTest
[junit4:junit4] Completed on J1 in 0.81s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.search.similarities.TestIBSimilarityFactory
[junit4:junit4] Completed on J1 in 0.15s, 2 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.search.ReturnFieldsTest
[junit4:junit4] Completed on J1 in 0.91s, 10 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.cloud.ZkCLITest
[junit4:junit4] Completed on J1 in 2.54s, 3 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.core.RAMDirectoryFactoryTest
[junit4:junit4] Completed on J1 in 0.01s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.search.TestFastLRUCache
[junit4:junit4] Completed on J1 in 0.13s, 7 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestWikipediaTokenizerFactory
[junit4:junit4] Completed on J1 in 0.02s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestHindiFilters
[junit4:junit4] Completed on J1 in 0.03s, 3 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.analysis.TestPorterStemFilterFactory
[junit4:junit4] Completed on J1 in 0.01s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.handler.component.SpellCheckComponentTest
[junit4:junit4] Completed on J1 in 8.25s, 9 tests
[junit4:junit4]  
[junit4:junit4] HEARTBEAT J1: 2012-07-25T09:40:34, no events in: 65.8s, approx. at: FullSolrCloudDistribCmdsTest.testDistribSearch
[junit4:junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4:junit4] Completed on J1 in 126.87s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.cloud.LeaderElectionIntegrationTest
[junit4:junit4] Completed on J1 in 27.65s, 2 tests
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.handler.component.DistributedSpellCheckComponentTest
[junit4:junit4] Completed on J1 in 12.62s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.TestDistributedGrouping
[junit4:junit4] Completed on J1 in 20.62s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.TestRandomFaceting
[junit4:junit4] Completed on J1 in 75.33s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.cloud.CloudStateUpdateTest
[junit4:junit4] Completed on J1 in 8.92s, 1 test
[junit4:junit4]  
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4] FAILURE  442s J0 | RecoveryZkTest.testDistribSearch
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([3D6B3CDC778FA2A:82303DD5B0279A16]:0)
[junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > 	at org.apache.solr.cloud.AbstractDistributedZkTestCase.waitForRecoveriesToFinish(AbstractDistributedZkTestCase.java:153)
[junit4:junit4]    > 	at org.apache.solr.cloud.AbstractDistributedZkTestCase.waitForRecoveriesToFinish(AbstractDistributedZkTestCase.java:115)
[junit4:junit4]    > 	at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:87)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:679)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:601)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:818)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:877)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:891)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:825)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(RandomizedRunner.java:132)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(RandomizedRunner.java:671)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:697)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:736)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:747)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551)
[junit4:junit4]    > 
[junit4:junit4]   2> 3 T1802 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1343209015356
[junit4:junit4]   2> 3 T1802 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 6 T1803 oazs.ZooKeeperServer.setTickTime tickTime set to 3000
[junit4:junit4]   2> 6 T1803 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 8 T1803 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 105 T1802 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40941 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@241b36ab
[junit4:junit4]   2> 106 T1808 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:40941
[junit4:junit4]   2> 106 T1808 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:40941, initiating session
[junit4:junit4]   2> 106 T1804 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:40942
[junit4:junit4]   2> 107 T1804 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:40942
[junit4:junit4]   2> 107 T1806 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 109 T1806 oazs.NIOServerCnxn.finishSessionInit Established session 0x138bd7f08410000 with negotiated timeout 10000 for client /140.211.11.196:40942
[junit4:junit4]   2> 109 T1808 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:40941, sessionid = 0x138bd7f08410000, negotiated timeout = 10000
[junit4:junit4]   2> 109 T1809 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@241b36ab name:ZooKeeperConnection Watcher:127.0.0.1:40941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 109 T1802 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 111 T1807 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x138bd7f08410000
[junit4:junit4]   2> 120 T1809 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 120 T1804 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:40942 which had sessionid 0x138bd7f08410000
[junit4:junit4]   2> 120 T1802 oaz.ZooKeeper.close Session: 0x138bd7f08410000 closed
[junit4:junit4]   2> 120 T1802 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40941/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@48c88966
[junit4:junit4]   2> 121 T1810 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:40941
[junit4:junit4]   2> 121 T1810 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:40941, initiating session
[junit4:junit4]   2> 122 T1804 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:40943
[junit4:junit4]   2> 122 T1804 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:40943
[junit4:junit4]   2> 123 T1806 oazs.NIOServerCnxn.finishSessionInit Established session 0x138bd7f08410001 with negotiated timeout 10000 for client /140.211.11.196:40943
[junit4:junit4]   2> 123 T1810 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:40941, sessionid = 0x138bd7f08410001, negotiated timeout = 10000
[junit4:junit4]   2> 123 T1811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48c88966 name:ZooKeeperConnection Watcher:127.0.0.1:40941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 123 T1802 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 133 T1802 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 135 T1802 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 137 T1802 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 139 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 143 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 247 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 249 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 251 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 264 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 266 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 273 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 275 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 277 T1802 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 285 T1807 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x138bd7f08410001
[junit4:junit4]   2> 286 T1811 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 286 T1802 oaz.ZooKeeper.close Session: 0x138bd7f08410001 closed
[junit4:junit4]   2> 286 T1804 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:40943 which had sessionid 0x138bd7f08410001
[junit4:junit4]   2> 287 T1802 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 289 T1802 oejs.AbstractConnector.doStart Started SocketConnector@0.0.0.0:40944
[junit4:junit4]   2> 289 T1802 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 290 T1802 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 290 T1802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 296 T1802 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 296 T1802 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 296 T1802 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 297 T1802 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 297 T1802 oasc.CoreContainer.<init> New CoreContainer 376448943
[junit4:junit4]   2> 297 T1802 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 298 T1802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 309 T1802 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 343 T1802 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:40941/solr
[junit4:junit4]   2> 344 T1802 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40941/solr sessionTimeout=12000 watcher=org.apache.solr.common.cloud.ConnectionManager@67c9a479
[junit4:junit4]   2> 344 T1821 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:40941
[junit4:junit4]   2> 346 T1821 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:40941, initiating session
[junit4:junit4]   2> 346 T1804 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:40945
[junit4:junit4]   2> 346 T1804 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:40945
[junit4:junit4]   2> 348 T1806 oazs.NIOServerCnxn.finishSessionInit Established session 0x138bd7f08410002 with negotiated timeout 12000 for client /140.211.11.196:40945
[junit4:junit4]   2> 348 T1821 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:40941, sessionid = 0x138bd7f08410002, negotiated timeout = 12000
[junit4:junit4]   2> 348 T1822 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67c9a479 name:ZooKeeperConnection Watcher:127.0.0.1:40941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 349 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 359 T1802 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 361 T1802 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40944_solr
[junit4:junit4]   2> 361 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:40944_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:40944_solr
[junit4:junit4]   2> 362 T1802 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40944_solr
[junit4:junit4]   2> 372 T1802 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 372 T1802 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 373 T1802 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 373 T1802 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 373 T1802 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 374 T1802 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 374 T1802 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 374 T1802 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 375 T1802 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 375 T1802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 387 T1802 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 391 T1802 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 403 T1802 oasc.Overseer.<init> Overseer (id=88028546030698498-127.0.0.1:40944_solr-n_0000000000) starting
[junit4:junit4]   2> 404 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410002 type:create cxid:0x19 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 405 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 406 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 415 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 416 T1824 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 417 T1802 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 420 T1802 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 427 T1802 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 427 T1802 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 428 T1823 oasc.Overseer$CloudStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 428 T1802 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 428 T1802 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 429 T1802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 430 T1802 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 430 T1802 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 472 T1802 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 555 T1802 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 657 T1802 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 667 T1802 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1360 T1802 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1367 T1802 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1370 T1802 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1377 T1802 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1381 T1802 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1383 T1802 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1384 T1802 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1385 T1802 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1343209015356/control/data/
[junit4:junit4]   2> 1385 T1802 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41df8a9c
[junit4:junit4]   2> 1386 T1802 oasc.SolrCore.getNewIndexDir WARNING New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1343209015356/control/data/index/
[junit4:junit4]   2> 1387 T1802 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1343209015356/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1417 T1802 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-1343209015356/control/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36f38d39; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1417 T1802 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1417 T1802 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 1418 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1419 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 1419 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1419 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 1420 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1420 T1802 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1420 T1802 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1421 T1802 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1421 T1802 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1421 T1802 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 1421 T1802 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 1422 T1802 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1422 T1802 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 1422 T1802 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 1423 T1802 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 1423 T1802 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1423 T1802 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1423 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1424 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1424 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1424 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1425 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1425 T1802 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1425 T1802 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 1426 T1802 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 1427 T1802 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1428 T1802 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 1429 T1802 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 1431 T1802 oass.SolrIndexSearcher.<init> Opening Searcher@cb17696 main
[junit4:junit4]   2> 1432 T1802 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1432 T1802 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1432 T1802 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 1437 T1802 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 1480 T1825 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@cb17696 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1481 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410002 type:create cxid:0x55 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 1489 T1802 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1489 T1802 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:40944/solr shardId:control_shard
[junit4:junit4]   2> 1490 T1802 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 1511 T1802 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 1522 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410002 type:create cxid:0x6c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1648 T1789 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> 1931 T1823 oascc.ZkStateReader.updateCloudState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1933 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410002 type:create cxid:0x77 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2130 T1822 oascc.ZkStateReader$2.process A cluster state change has occurred
[junit4:junit4]   2> 2186 T1802 oasc.ZkController.register We are http://127.0.0.1:40944/solr/collection1/ and leader is http://127.0.0.1:40944/solr/collection1/
[junit4:junit4]   2> 2186 T1802 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40944/solr
[junit4:junit4]   2> 2186 T1802 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 2189 T1802 oascc.ZkStateReader.updateCloudState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2189 T1802 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2190 T1802 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2190 T1802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2198 T1802 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2200 T1802 oejs.AbstractConnector.doStart Started SocketConnector@0.0.0.0:40947
[junit4:junit4]   2> 2201 T1802 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2201 T1802 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2201 T1802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2206 T1802 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2206 T1802 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2206 T1802 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2206 T1802 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2207 T1802 oasc.CoreContainer.<init> New CoreContainer 2096048678
[junit4:junit4]   2> 2207 T1802 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2207 T1802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2215 T1802 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2226 T1802 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:40941/solr
[junit4:junit4]   2> 2227 T1802 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:40941/solr sessionTimeout=12000 watcher=org.apache.solr.common.cloud.ConnectionManager@39b864de
[junit4:junit4]   2> 2228 T1835 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:40941
[junit4:junit4]   2> 2228 T1835 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:40941, initiating session
[junit4:junit4]   2> 2228 T1804 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:40948
[junit4:junit4]   2> 2229 T1804 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:40948
[junit4:junit4]   2> 2229 T1806 oazs.NIOServerCnxn.finishSessionInit Established session 0x138bd7f08410003 with negotiated timeout 12000 for client /140.211.11.196:40948
[junit4:junit4]   2> 2230 T1835 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:40941, sessionid = 0x138bd7f08410003, negotiated timeout = 12000
[junit4:junit4]   2> 2230 T1836 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39b864de name:ZooKeeperConnection Watcher:127.0.0.1:40941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2230 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2231 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2246 T1802 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40947_solr
[junit4:junit4]   2> 2246 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:40947_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:40947_solr
[junit4:junit4]   2> 2247 T1802 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40947_solr
[junit4:junit4]   2> 2248 T1822 oascc.ZkStateReader$3.process Updating live nodes
[junit4:junit4]   2> 2249 T1802 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2249 T1802 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2249 T1802 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2249 T1802 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2249 T1802 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2250 T1802 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2250 T1802 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2250 T1802 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2250 T1802 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2251 T1802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2260 T1802 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2265 T1802 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2265 T1802 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 2265 T1802 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2266 T1802 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 2266 T1802 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2267 T1802 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2267 T1802 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2294 T1802 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 2365 T1802 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2466 T1802 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2470 T1802 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2663 T1823 oascc.ZkStateReader.updateCloudState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2680 T1822 oascc.ZkStateReader$2.process A cluster state change has occurred
[junit4:junit4]   2> 2680 T1836 oascc.ZkStateReader$2.process A cluster state change has occurred
[junit4:junit4]   2> 2921 T1802 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2928 T1802 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2931 T1802 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2939 T1802 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2942 T1802 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2945 T1802 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2946 T1802 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2947 T1802 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1343209015356/jetty1/
[junit4:junit4]   2> 2947 T1802 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@41df8a9c
[junit4:junit4]   2> 2948 T1802 oasc.SolrCore.getNewIndexDir WARNING New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1343209015356/jetty1/index/
[junit4:junit4]   2> 2948 T1802 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1343209015356/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2951 T1802 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-1343209015356/jetty1/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6cd7ae93; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2952 T1802 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2952 T1802 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2953 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2954 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2954 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2954 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2955 T1802 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2955 T1802 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2955 T1802 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2956 T1802 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2956 T1802 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2956 T1802 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2957 T1802 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2957 T1802 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2957 T1802 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2958 T1802 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2958 T1802 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2958 T1802 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2959 T1802 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2959 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2959 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2959 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2960 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2960 T1802 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2960 T1802 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2961 T1802 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2961 T1802 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 2962 T1802 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2964 T1802 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2964 T1802 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 2967 T1802 oass.SolrIndexSearcher.<init> Opening Searcher@5f3f8570 main
[junit4:junit4]   2> 2967 T1802 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2967 T1802 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2968 T1802 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 2972 T1802 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 3016 T1837 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f3f8570 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3184 T1823 oascc.ZkStateReader.updateCloudState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3197 T1822 oascc.ZkStateReader$2.process A cluster state change has occurred
[junit4:junit4]   2> 3197 T1836 oascc.ZkStateReader$2.process A cluster state change has occurred
[junit4:junit4]   2> 3519 T1802 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3519 T1802 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:40947/solr shardId:shard1
[junit4:junit4]   2> 3520 T1802 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 3534 T1802 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 3537 T1807 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x138bd7f08410003 type:create cxid:0x4b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3699 T1823 oascc.ZkStateReader.updateCloudState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3715 T1822 oascc.ZkStateReader$2.process A cluster state change has occurred
[junit4:junit4]   2> 3715 T1836 oascc.ZkStateReader$2.process A cluster state change has occurred
[junit4:junit4]   2> 3743 T1802 oasc.ZkController.register We are http://127.0.0.1:40947/solr/collection1/ and leader is http://127.0.0.1:40947/solr/collection1/
[junit4:junit4]   2> 3743 T1802 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40947/solr
[junit4:junit4]   2> 3743 T1802 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3745 T1802 oascc.ZkStateReader.updateCloudState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3746 T1802 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3746 T1802 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3746 T1802 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3758 T1802 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3761 T1802 oejs.AbstractConnector.doStart Started SocketConnector@0.0.0.0:40949
[...truncated 145363 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/build.xml:29: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build.xml:144: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/lucene/module-build.xml:62: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/lucene/common-build.xml:1094: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/lucene/common-build.xml:789: There were test failures: 297 suites, 1055 tests, 1 suite-level error, 1 failure, 5 ignored

Total time: 33 minutes 50 seconds
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message