Return-Path: 
 <dev-return-113034-apmail-lucene-dev-archive=lucene.apache.org@lucene.apache.org>
X-Original-To: apmail-lucene-dev-archive@www.apache.org
Delivered-To: apmail-lucene-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 31DFFD51B
	for <apmail-lucene-dev-archive@www.apache.org>;
 Mon, 29 Oct 2012 01:08:36 +0000 (UTC)
Received: (qmail 13883 invoked by uid 500); 29 Oct 2012 01:08:34 -0000
Delivered-To: apmail-lucene-dev-archive@lucene.apache.org
Received: (qmail 13829 invoked by uid 500); 29 Oct 2012 01:08:34 -0000
Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@lucene.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@lucene.apache.org>
List-Post: <mailto:dev@lucene.apache.org>
List-Id: <dev.lucene.apache.org>
Reply-To: dev@lucene.apache.org
Delivered-To: mailing list dev@lucene.apache.org
Received: (qmail 13822 invoked by uid 99); 29 Oct 2012 01:08:34 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Oct 2012 01:08:34 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of markrmiller@gmail.com
 designates 209.85.216.176 as permitted sender)
Received: from [209.85.216.176] (HELO mail-qc0-f176.google.com)
 (209.85.216.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Oct 2012 01:08:26 +0000
Received: by mail-qc0-f176.google.com with SMTP id n41so2630526qco.35
        for <dev@lucene.apache.org>; Sun, 28 Oct 2012 18:08:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type;
        bh=ugE7byfTZx2RVNboE0WJwj3nzYimP8zV4C/NNcydi2A=;
        b=SO45JsXlMtMeLOgQiZiuWrT3cccg4+FX0KQW/6YzqVitYaXGM8KW/3wfdeUTlUziZJ
         8OOHI6Sjx0TRydkpMiTIOakXk3NemtO3jcDop9aaU4CJAgy6v7Mm0DuduuiIc1vh3jxD
         wfsNPLvjM4PySDUgjcGq/yb8KpgH+F8SimA9M/zF4sLnYCLZvmh3LIpccclnuLiBvJvl
         FBSM5f5W7u1jKkVvrDnCMkFInE+6SqHkoUFNWQUrz5O85naFEhunCmTqTBANyUHiytNj
         rHpUaRzf5+36XSMK+W1jc5s+tasBYF39HxI80Nkk0vL2ueX/WE5YxjrOeFi453LMprQU
         Kd9Q==
Received: by 10.49.71.71 with SMTP id s7mr20667299qeu.33.1351472885054;
        Sun, 28 Oct 2012 18:08:05 -0700 (PDT)
Received: from [192.168.1.230] (ool-18bf2931.dyn.optonline.net.
 [24.191.41.49])
        by mx.google.com with ESMTPS id t14sm4364303qef.3.2012.10.28.18.07.40
        (version=SSLv3 cipher=OTHER);
        Sun, 28 Oct 2012 18:08:01 -0700 (PDT)
Message-ID: <508DD6E2.8020907@gmail.com>
Date: Sun, 28 Oct 2012 21:07:46 -0400
From: Mark Miller <markrmiller@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:16.0) Gecko/20121017 Thunderbird/16.0.1
MIME-Version: 1.0
To: dev@lucene.apache.org
Subject: Re: Build failed in Jenkins: fast-io-beasting #6191
References: <1133952077.23.1351472730817.JavaMail.rmuir@beast>
In-Reply-To: <1133952077.23.1351472730817.JavaMail.rmuir@beast>
Content-Type: multipart/alternative;
 boundary="------------030108000501080704080901"
X-Virus-Checked: Checked by ClamAV on apache.org

--------------030108000501080704080901
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

So the issue here appears to be:

[junit4:junit4]   2> 72153 T359 oasc.SolrException.log SEVERE null:org.apache.solr.common.SolrException: Could not get shard_id for core: collection1
[junit4:junit4]   2> 		at org.apache.solr.cloud.ZkController.doGetShardIdProcess(ZkController.java:996)
[junit4:junit4]   2> 		at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1054)
[junit4:junit4]   2> 		at org.apache.solr.core.CoreContainer.register(CoreContainer.java:657)
[junit4:junit4]   2> 		at org.apache.solr.core.CoreContainer.load(CoreContainer.java:530)
[junit4:junit4]   2> 		at org.apache.solr.core.CoreContainer.load(CoreContainer.java:351)

Sami, any chance you are around to look into this?

- Mark



On 10/28/2012 09:05 PM, Charlie Cron wrote:
> See <http://sierranevada.servebeer.com/job/fast-io-beasting/6191/>
>
> ------------------------------------------
> [...truncated 13491 lines...]
> [junit4:junit4]   2> 89052 T359 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
> [junit4:junit4]   2> 89053 T359 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
> [junit4:junit4]   2> 89055 T432 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
> [junit4:junit4]   2> 89055 T432 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aaa09ba0c0006 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> [junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> [junit4:junit4]   2>
> [junit4:junit4]   2> 89347 T404 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
> [junit4:junit4]   2> 89368 T418 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
> [junit4:junit4]   2> 89368 T418 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aaa09ba0c0005 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> [junit4:junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
> [junit4:junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
> [junit4:junit4]   2>
> [junit4:junit4]   2> 89448 T405 oaz.ClientCnxn$EventThread.run EventThread shut down
> [junit4:junit4]   2> 89448 T359 oaz.ZooKeeper.close Session: 0x13aaa09ba0c0004 closed
> [junit4:junit4]   2> 89471 T359 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
> [junit4:junit4]   2> 89522 T359 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 38259
> [junit4:junit4]   2> 89522 T359 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=711560962
> [junit4:junit4]   2> 89523 T359 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7402a821
> [junit4:junit4]   2> 89528 T359 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
> [junit4:junit4]   2> 89529 T359 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
> [junit4:junit4]   2> 89529 T359 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
> [junit4:junit4]   2> 89530 T359 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
> [junit4:junit4]   2> 89531 T359 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
> [junit4:junit4]   2> 90655 T418 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
> [junit4:junit4]   2> 90756 T419 oaz.ClientCnxn$EventThread.run EventThread shut down
> [junit4:junit4]   2> 90756 T359 oaz.ZooKeeper.close Session: 0x13aaa09ba0c0005 closed
> [junit4:junit4]   2> 90779 T359 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
> [junit4:junit4]   2> 90831 T359 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50727
> [junit4:junit4]   2> 90831 T359 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2007775322
> [junit4:junit4]   2> 90853 T432 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
> [junit4:junit4]   2> 90953 T433 oaz.ClientCnxn$EventThread.run EventThread shut down
> [junit4:junit4]   2> 90953 T359 oaz.ZooKeeper.close Session: 0x13aaa09ba0c0006 closed
> [junit4:junit4]   2> 90976 T359 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
> [junit4:junit4]   2> 91043 T359 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
> [junit4:junit4]   1>        /solr/collections/control_collection/leader_elect/control_shard/election/88570101439922178-127.0.0.1:49583_solr_collection1-n_0000000000 (0)
> [junit4:junit4]   1>     /solr/collections/control_collection/leaders (1)
> [junit4:junit4]   1>      /solr/collections/control_collection/leaders/control_shard (0)
> [junit4:junit4]   1>      DATA:
> [junit4:junit4]   1>          {
> [junit4:junit4]   1>            "core":"collection1",
> [junit4:junit4]   1>            "node_name":"127.0.0.1:49583_solr",
> [junit4:junit4]   1>            "base_url":"http://127.0.0.1:49583/solr"}
> [junit4:junit4]   1>   /solr/clusterstate.json (0)
> [junit4:junit4]   1>   DATA:
> [junit4:junit4]   1>       {
> [junit4:junit4]   1>         "collection1":{"shard1":{
> [junit4:junit4]   1>             "range":"80000000-7fffffff",
> [junit4:junit4]   1>             "replicas":{
> [junit4:junit4]   1>               "127.0.0.1:39354_solr_collection1":{
> [junit4:junit4]   1>                 "shard":"shard1",
> [junit4:junit4]   1>                 "roles":null,
> [junit4:junit4]   1>                 "state":"active",
> [junit4:junit4]   1>                 "core":"collection1",
> [junit4:junit4]   1>                 "collection":"collection1",
> [junit4:junit4]   1>                 "node_name":"127.0.0.1:39354_solr",
> [junit4:junit4]   1>                 "base_url":"http://127.0.0.1:39354/solr",
> [junit4:junit4]   1>                 "leader":"true"},
> [junit4:junit4]   1>               "127.0.0.1:51265_solr_collection1":{
> [junit4:junit4]   1>                 "shard":"shard1",
> [junit4:junit4]   1>                 "roles":null,
> [junit4:junit4]   1>                 "state":"active",
> [junit4:junit4]   1>                 "core":"collection1",
> [junit4:junit4]   1>                 "collection":"collection1",
> [junit4:junit4]   1>                 "node_name":"127.0.0.1:51265_solr",
> [junit4:junit4]   1>                 "base_url":"http://127.0.0.1:51265/solr"},
> [junit4:junit4]   1>               "127.0.0.1:38259_solr_collection1":{
> [junit4:junit4]   1>                 "shard":"shard1",
> [junit4:junit4]   1>                 "roles":null,
> [junit4:junit4]   1>                 "state":"active",
> [junit4:junit4]   1>                 "core":"collection1",
> [junit4:junit4]   1>                 "collection":"collection1",
> [junit4:junit4]   1>                 "node_name":"127.0.0.1:38259_solr",
> [junit4:junit4]   1>                 "base_url":"http://127.0.0.1:38259/solr"},
> [junit4:junit4]   1>               "127.0.0.1:50727_solr_collection1":{
> [junit4:junit4]   1>                 "shard":null,
> [junit4:junit4]   1>                 "roles":null,
> [junit4:junit4]   1>                 "state":"down",
> [junit4:junit4]   1>                 "core":"collection1",
> [junit4:junit4]   1>                 "collection":"collection1",
> [junit4:junit4]   1>                 "node_name":"127.0.0.1:50727_solr",
> [junit4:junit4]   1>                 "base_url":"http://127.0.0.1:50727/solr"}}}},
> [junit4:junit4]   1>         "control_collection":{"control_shard":{"replicas":{"127.0.0.1:49583_solr_collection1":{
> [junit4:junit4]   1>                 "shard":"control_shard",
> [junit4:junit4]   1>                 "roles":null,
> [junit4:junit4]   1>                 "state":"active",
> [junit4:junit4]   1>                 "core":"collection1",
> [junit4:junit4]   1>                 "collection":"control_collection",
> [junit4:junit4]   1>                 "node_name":"127.0.0.1:49583_solr",
> [junit4:junit4]   1>                 "base_url":"http://127.0.0.1:49583/solr",
> [junit4:junit4]   1>                 "leader":"true"}}}}}
> [junit4:junit4]   1>  /zookeeper (1)
> [junit4:junit4]   1>  DATA:
> [junit4:junit4]   1>
> [junit4:junit4]   1>
> [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=FE6D4DCD35D078A1 -Dtests.slow=true -Dtests.locale=ar_SA -Dtests.timezone=Asia/Bishkek -Dtests.file.encoding=UTF-8
> [junit4:junit4] FAILURE 91.1s J5 | SyncSliceTest.testDistribSearch <<<
> [junit4:junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 15 seconds
> [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([FE6D4DCD35D078A1:7F8BC3D5428F189D]:0)
> [junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
> [junit4:junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:165)
> [junit4:junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:558)
> [junit4:junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1133)
> [junit4:junit4]    > 	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:89)
> [junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:695)
> [junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [junit4:junit4]    > 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> [junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> [junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:597)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> [junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]   2> 91065 T359 oas.SolrTestCaseJ4.deleteCore ###deleteCore
> [junit4:junit4]   2> 212078 T359 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE ERROR: SolrIndexSearcher opens=8 closes=7
> [junit4:junit4]   2> 212082 T358 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
> [junit4:junit4]   2> 232141 T358 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.SyncSliceTest:
> [junit4:junit4]   2> 	   1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
> [junit4:junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
> [junit4:junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> [junit4:junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> [junit4:junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> [junit4:junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> [junit4:junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> [junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]   2> 232142 T358 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
> [junit4:junit4]   2> 	   1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
> [junit4:junit4]   2> 235144 T358 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
> [junit4:junit4]   2> 	   1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
> [junit4:junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
> [junit4:junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> [junit4:junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> [junit4:junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> [junit4:junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> [junit4:junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> [junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=ar_SA, timezone=Asia/Bishkek
> [junit4:junit4]   2> NOTE: Linux 3.2.0-24-generic amd64/Sun Microsystems Inc. 1.6.0_27 (64-bit)/cpus=8,threads=2,free=150778856,total=311099392
> [junit4:junit4]   2> NOTE: All tests run in this JVM: [TestLFUCache, TestRecovery, MBeansHandlerTest, SolrRequestParserTest, TestFiltering, TestDefaultSimilarityFactory, TestQueryTypes, TestPerFieldSimilarity, TestReplicationHandler, NotRequiredUniqueKeyTest, TestDocumentBuilder, SpellingQueryConverterTest, BadComponentTest, QueryEqualityTest, DefaultValueUpdateProcessorTest, SimpleFacetsTest, DirectSolrSpellCheckerTest, QueryElevationComponentTest, TestRemoteStreaming, DocumentBuilderTest, StandardRequestHandlerTest, DirectUpdateHandlerOptimizeTest, TestArbitraryIndexDir, SolrCoreCheckLockOnStartupTest, AlternateDirectoryTest, MultiTermTest, TestLMJelinekMercerSimilarityFactory, OpenExchangeRatesOrgProviderTest, UUIDFieldTest, SyncSliceTest]
> [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.seed=FE6D4DCD35D078A1 -Dtests.slow=true -Dtests.locale=ar_SA -Dtests.timezone=Asia/Bishkek -Dtests.file.encoding=UTF-8
> [junit4:junit4] ERROR   0.00s J5 | SyncSliceTest (suite) <<<
> [junit4:junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=8 closes=7
> [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([FE6D4DCD35D078A1]:0)
> [junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
> [junit4:junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:242)
> [junit4:junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:97)
> [junit4:junit4]    > 	at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> [junit4:junit4]    > 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> [junit4:junit4]    > 	at java.lang.reflect.Method.invoke(Method.java:597)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> [junit4:junit4]    > 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> [junit4:junit4]    > 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> [junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> [junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.SyncSliceTest:
> [junit4:junit4]    >    1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
> [junit4:junit4]    >         at sun.misc.Unsafe.park(Native Method)
> [junit4:junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> [junit4:junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> [junit4:junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> [junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> [junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> [junit4:junit4]    >         at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([FE6D4DCD35D078A1]:0)
> [junit4:junit4]    > Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
> [junit4:junit4]    >    1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
> [junit4:junit4]    >         at sun.misc.Unsafe.park(Native Method)
> [junit4:junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> [junit4:junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> [junit4:junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> [junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> [junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> [junit4:junit4]    >         at java.lang.Thread.run(Thread.java:662)
> [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([FE6D4DCD35D078A1]:0)
> [junit4:junit4] Completed on J5 in 235.18s, 1 test, 2 failures, 2 errors <<< FAILURES!
> [junit4:junit4]
> [junit4:junit4]
> [junit4:junit4] Tests with failures:
> [junit4:junit4]   - org.apache.solr.cloud.SyncSliceTest.testDistribSearch
> [junit4:junit4]   - org.apache.solr.cloud.SyncSliceTest (suite)
> [junit4:junit4]
> [junit4:junit4]
> [junit4:junit4] JVM J0:     0.94 ..   167.39 =   166.45s
> [junit4:junit4] JVM J1:     0.97 ..   125.55 =   124.58s
> [junit4:junit4] JVM J2:     0.94 ..   126.94 =   125.99s
> [junit4:junit4] JVM J3:     0.94 ..   124.33 =   123.39s
> [junit4:junit4] JVM J4:     0.94 ..   131.56 =   130.61s
> [junit4:junit4] JVM J5:     1.22 ..   348.93 =   347.71s
> [junit4:junit4] JVM J6:     0.97 ..   130.46 =   129.49s
> [junit4:junit4] JVM J7:     0.97 ..   118.79 =   117.82s
> [junit4:junit4] Execution time total: 5 minutes 48 seconds
> [junit4:junit4] Tests summary: 233 suites, 965 tests, 3 suite-level errors, 1 failure, 7 ignored (2 assumptions)
>
> BUILD FAILED
> <http://sierranevada.servebeer.com/job/fast-io-beasting/ws/build.xml>:39: The following error occurred while executing this line:
> <http://sierranevada.servebeer.com/job/fast-io-beasting/ws/solr/build.xml>:178: The following error occurred while executing this line:
> <http://sierranevada.servebeer.com/job/fast-io-beasting/ws/lucene/module-build.xml>:61: The following error occurred while executing this line:
> <http://sierranevada.servebeer.com/job/fast-io-beasting/ws/lucene/common-build.xml>:1159: The following error occurred while executing this line:
> <http://sierranevada.servebeer.com/job/fast-io-beasting/ws/lucene/common-build.xml>:827: There were test failures: 233 suites, 965 tests, 3 suite-level errors, 1 failure, 7 ignored (2 assumptions)
>
> Total time: 10 minutes 52 seconds
> Build step 'Invoke Ant' marked build as failure
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>


--------------030108000501080704080901
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">So the issue here appears to be:<br>
      <br>
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <pre style="white-space: pre-wrap; word-wrap: break-word; margin: 0px; color: rgb(0, 0, 0); font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); ">[junit4:junit4]   2&gt; 72153 T359 oasc.SolrException.log SEVERE null:org.apache.solr.common.SolrException: Could not get shard_id for core: collection1
[junit4:junit4]   2&gt; 		at org.apache.solr.cloud.ZkController.doGetShardIdProcess(ZkController.java:996)
[junit4:junit4]   2&gt; 		at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1054)
[junit4:junit4]   2&gt; 		at org.apache.solr.core.CoreContainer.register(CoreContainer.java:657)
[junit4:junit4]   2&gt; 		at org.apache.solr.core.CoreContainer.load(CoreContainer.java:530)
[junit4:junit4]   2&gt; 		at org.apache.solr.core.CoreContainer.load(CoreContainer.java:351)

Sami, any chance you are around to look into this?

- Mark
</pre>
      <br>
      <br>
      On 10/28/2012 09:05 PM, Charlie Cron wrote:<br>
    </div>
    <blockquote
      cite="mid:1133952077.23.1351472730817.JavaMail.rmuir@beast"
      type="cite">
      <pre wrap="">See <a class="moz-txt-link-rfc2396E" href="http://sierranevada.servebeer.com/job/fast-io-beasting/6191/">&lt;http://sierranevada.servebeer.com/job/fast-io-beasting/6191/&gt;</a>

------------------------------------------
[...truncated 13491 lines...]
[junit4:junit4]   2&gt; 89052 T359 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2&gt; 89053 T359 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2&gt; 89055 T432 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
[junit4:junit4]   2&gt; 89055 T432 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aaa09ba0c0006 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4:junit4]   2&gt; 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4:junit4]   2&gt; 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
[junit4:junit4]   2&gt; 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
[junit4:junit4]   2&gt; 
[junit4:junit4]   2&gt; 89347 T404 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
[junit4:junit4]   2&gt; 89368 T418 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
[junit4:junit4]   2&gt; 89368 T418 oaz.ClientCnxn$SendThread.run WARNING Session 0x13aaa09ba0c0005 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
[junit4:junit4]   2&gt; 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit4:junit4]   2&gt; 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
[junit4:junit4]   2&gt; 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1143)
[junit4:junit4]   2&gt; 
[junit4:junit4]   2&gt; 89448 T405 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2&gt; 89448 T359 oaz.ZooKeeper.close Session: 0x13aaa09ba0c0004 closed
[junit4:junit4]   2&gt; 89471 T359 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2&gt; 89522 T359 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 38259
[junit4:junit4]   2&gt; 89522 T359 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=711560962
[junit4:junit4]   2&gt; 89523 T359 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7402a821
[junit4:junit4]   2&gt; 89528 T359 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2&gt; 89529 T359 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2&gt; 89529 T359 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2&gt; 89530 T359 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2&gt; 89531 T359 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2&gt; 90655 T418 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
[junit4:junit4]   2&gt; 90756 T419 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2&gt; 90756 T359 oaz.ZooKeeper.close Session: 0x13aaa09ba0c0005 closed
[junit4:junit4]   2&gt; 90779 T359 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2&gt; 90831 T359 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 50727
[junit4:junit4]   2&gt; 90831 T359 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2007775322
[junit4:junit4]   2&gt; 90853 T432 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost.localdomain/127.0.0.1:34975
[junit4:junit4]   2&gt; 90953 T433 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2&gt; 90953 T359 oaz.ZooKeeper.close Session: 0x13aaa09ba0c0006 closed
[junit4:junit4]   2&gt; 90976 T359 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2&gt; 91043 T359 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   1&gt;        /solr/collections/control_collection/leader_elect/control_shard/election/88570101439922178-127.0.0.1:49583_solr_collection1-n_0000000000 (0)
[junit4:junit4]   1&gt;     /solr/collections/control_collection/leaders (1)
[junit4:junit4]   1&gt;      /solr/collections/control_collection/leaders/control_shard (0)
[junit4:junit4]   1&gt;      DATA:
[junit4:junit4]   1&gt;          {
[junit4:junit4]   1&gt;            "core":"collection1",
[junit4:junit4]   1&gt;            "node_name":"127.0.0.1:49583_solr",
[junit4:junit4]   1&gt;            "base_url":<a class="moz-txt-link-rfc2396E" href="http://127.0.0.1:49583/solr">"http://127.0.0.1:49583/solr"</a>}
[junit4:junit4]   1&gt;   /solr/clusterstate.json (0)
[junit4:junit4]   1&gt;   DATA:
[junit4:junit4]   1&gt;       {
[junit4:junit4]   1&gt;         "collection1":{"shard1":{
[junit4:junit4]   1&gt;             "range":"80000000-7fffffff",
[junit4:junit4]   1&gt;             "replicas":{
[junit4:junit4]   1&gt;               "127.0.0.1:39354_solr_collection1":{
[junit4:junit4]   1&gt;                 "shard":"shard1",
[junit4:junit4]   1&gt;                 "roles":null,
[junit4:junit4]   1&gt;                 "state":"active",
[junit4:junit4]   1&gt;                 "core":"collection1",
[junit4:junit4]   1&gt;                 "collection":"collection1",
[junit4:junit4]   1&gt;                 "node_name":"127.0.0.1:39354_solr",
[junit4:junit4]   1&gt;                 "base_url":<a class="moz-txt-link-rfc2396E" href="http://127.0.0.1:39354/solr">"http://127.0.0.1:39354/solr"</a>,
[junit4:junit4]   1&gt;                 "leader":"true"},
[junit4:junit4]   1&gt;               "127.0.0.1:51265_solr_collection1":{
[junit4:junit4]   1&gt;                 "shard":"shard1",
[junit4:junit4]   1&gt;                 "roles":null,
[junit4:junit4]   1&gt;                 "state":"active",
[junit4:junit4]   1&gt;                 "core":"collection1",
[junit4:junit4]   1&gt;                 "collection":"collection1",
[junit4:junit4]   1&gt;                 "node_name":"127.0.0.1:51265_solr",
[junit4:junit4]   1&gt;                 "base_url":<a class="moz-txt-link-rfc2396E" href="http://127.0.0.1:51265/solr">"http://127.0.0.1:51265/solr"</a>},
[junit4:junit4]   1&gt;               "127.0.0.1:38259_solr_collection1":{
[junit4:junit4]   1&gt;                 "shard":"shard1",
[junit4:junit4]   1&gt;                 "roles":null,
[junit4:junit4]   1&gt;                 "state":"active",
[junit4:junit4]   1&gt;                 "core":"collection1",
[junit4:junit4]   1&gt;                 "collection":"collection1",
[junit4:junit4]   1&gt;                 "node_name":"127.0.0.1:38259_solr",
[junit4:junit4]   1&gt;                 "base_url":<a class="moz-txt-link-rfc2396E" href="http://127.0.0.1:38259/solr">"http://127.0.0.1:38259/solr"</a>},
[junit4:junit4]   1&gt;               "127.0.0.1:50727_solr_collection1":{
[junit4:junit4]   1&gt;                 "shard":null,
[junit4:junit4]   1&gt;                 "roles":null,
[junit4:junit4]   1&gt;                 "state":"down",
[junit4:junit4]   1&gt;                 "core":"collection1",
[junit4:junit4]   1&gt;                 "collection":"collection1",
[junit4:junit4]   1&gt;                 "node_name":"127.0.0.1:50727_solr",
[junit4:junit4]   1&gt;                 "base_url":<a class="moz-txt-link-rfc2396E" href="http://127.0.0.1:50727/solr">"http://127.0.0.1:50727/solr"</a>}}}},
[junit4:junit4]   1&gt;         "control_collection":{"control_shard":{"replicas":{"127.0.0.1:49583_solr_collection1":{
[junit4:junit4]   1&gt;                 "shard":"control_shard",
[junit4:junit4]   1&gt;                 "roles":null,
[junit4:junit4]   1&gt;                 "state":"active",
[junit4:junit4]   1&gt;                 "core":"collection1",
[junit4:junit4]   1&gt;                 "collection":"control_collection",
[junit4:junit4]   1&gt;                 "node_name":"127.0.0.1:49583_solr",
[junit4:junit4]   1&gt;                 "base_url":<a class="moz-txt-link-rfc2396E" href="http://127.0.0.1:49583/solr">"http://127.0.0.1:49583/solr"</a>,
[junit4:junit4]   1&gt;                 "leader":"true"}}}}}
[junit4:junit4]   1&gt;  /zookeeper (1)
[junit4:junit4]   1&gt;  DATA:
[junit4:junit4]   1&gt;      
[junit4:junit4]   1&gt; 
[junit4:junit4]   2&gt; NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=FE6D4DCD35D078A1 -Dtests.slow=true -Dtests.locale=ar_SA -Dtests.timezone=Asia/Bishkek -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 91.1s J5 | SyncSliceTest.testDistribSearch &lt;&lt;&lt;
[junit4:junit4]    &gt; Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 15 seconds
[junit4:junit4]    &gt; 	at __randomizedtesting.SeedInfo.seed([FE6D4DCD35D078A1:7F8BC3D5428F189D]:0)
[junit4:junit4]    &gt; 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    &gt; 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:165)
[junit4:junit4]    &gt; 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:558)
[junit4:junit4]    &gt; 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1133)
[junit4:junit4]    &gt; 	at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:89)
[junit4:junit4]    &gt; 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:695)
[junit4:junit4]    &gt; 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    &gt; 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit4:junit4]    &gt; 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4]    &gt; 	at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    &gt; 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2&gt; 91065 T359 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2&gt; 212078 T359 oas.SolrTestCaseJ4.endTrackingSearchers SEVERE ERROR: SolrIndexSearcher opens=8 closes=7
[junit4:junit4]   2&gt; 212082 T358 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4]   2&gt; 232141 T358 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.SyncSliceTest: 
[junit4:junit4]   2&gt; 	   1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
[junit4:junit4]   2&gt; 	        at sun.misc.Unsafe.park(Native Method)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4]   2&gt; 	        at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2&gt; 232142 T358 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2&gt; 	   1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
[junit4:junit4]   2&gt; 235144 T358 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2&gt; 	   1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
[junit4:junit4]   2&gt; 	        at sun.misc.Unsafe.park(Native Method)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4]   2&gt; 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4]   2&gt; 	        at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]   2&gt; NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=ar_SA, timezone=Asia/Bishkek
[junit4:junit4]   2&gt; NOTE: Linux 3.2.0-24-generic amd64/Sun Microsystems Inc. 1.6.0_27 (64-bit)/cpus=8,threads=2,free=150778856,total=311099392
[junit4:junit4]   2&gt; NOTE: All tests run in this JVM: [TestLFUCache, TestRecovery, MBeansHandlerTest, SolrRequestParserTest, TestFiltering, TestDefaultSimilarityFactory, TestQueryTypes, TestPerFieldSimilarity, TestReplicationHandler, NotRequiredUniqueKeyTest, TestDocumentBuilder, SpellingQueryConverterTest, BadComponentTest, QueryEqualityTest, DefaultValueUpdateProcessorTest, SimpleFacetsTest, DirectSolrSpellCheckerTest, QueryElevationComponentTest, TestRemoteStreaming, DocumentBuilderTest, StandardRequestHandlerTest, DirectUpdateHandlerOptimizeTest, TestArbitraryIndexDir, SolrCoreCheckLockOnStartupTest, AlternateDirectoryTest, MultiTermTest, TestLMJelinekMercerSimilarityFactory, OpenExchangeRatesOrgProviderTest, UUIDFieldTest, SyncSliceTest]
[junit4:junit4]   2&gt; NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.seed=FE6D4DCD35D078A1 -Dtests.slow=true -Dtests.locale=ar_SA -Dtests.timezone=Asia/Bishkek -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR   0.00s J5 | SyncSliceTest (suite) &lt;&lt;&lt;
[junit4:junit4]    &gt; Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=8 closes=7
[junit4:junit4]    &gt; 	at __randomizedtesting.SeedInfo.seed([FE6D4DCD35D078A1]:0)
[junit4:junit4]    &gt; 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    &gt; 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:242)
[junit4:junit4]    &gt; 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:97)
[junit4:junit4]    &gt; 	at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
[junit4:junit4]    &gt; 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit4:junit4]    &gt; 	at java.lang.reflect.Method.invoke(Method.java:597)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:700)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    &gt; 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    &gt; 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    &gt; 	at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]    &gt; Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.SyncSliceTest: 
[junit4:junit4]    &gt;    1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
[junit4:junit4]    &gt;         at sun.misc.Unsafe.park(Native Method)
[junit4:junit4]    &gt;         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
[junit4:junit4]    &gt;         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4]    &gt;         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4]    &gt;         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4]    &gt;         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4]    &gt;         at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]    &gt; 	at __randomizedtesting.SeedInfo.seed([FE6D4DCD35D078A1]:0)
[junit4:junit4]    &gt; Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    &gt;    1) Thread[id=434, name=searcherExecutor-405-thread-1, state=WAITING, group=TGRP-SyncSliceTest]
[junit4:junit4]    &gt;         at sun.misc.Unsafe.park(Native Method)
[junit4:junit4]    &gt;         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
[junit4:junit4]    &gt;         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
[junit4:junit4]    &gt;         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
[junit4:junit4]    &gt;         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
[junit4:junit4]    &gt;         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
[junit4:junit4]    &gt;         at java.lang.Thread.run(Thread.java:662)
[junit4:junit4]    &gt; 	at __randomizedtesting.SeedInfo.seed([FE6D4DCD35D078A1]:0)
[junit4:junit4] Completed on J5 in 235.18s, 1 test, 2 failures, 2 errors &lt;&lt;&lt; FAILURES!
[junit4:junit4] 
[junit4:junit4] 
[junit4:junit4] Tests with failures:
[junit4:junit4]   - org.apache.solr.cloud.SyncSliceTest.testDistribSearch
[junit4:junit4]   - org.apache.solr.cloud.SyncSliceTest (suite)
[junit4:junit4] 
[junit4:junit4] 
[junit4:junit4] JVM J0:     0.94 ..   167.39 =   166.45s
[junit4:junit4] JVM J1:     0.97 ..   125.55 =   124.58s
[junit4:junit4] JVM J2:     0.94 ..   126.94 =   125.99s
[junit4:junit4] JVM J3:     0.94 ..   124.33 =   123.39s
[junit4:junit4] JVM J4:     0.94 ..   131.56 =   130.61s
[junit4:junit4] JVM J5:     1.22 ..   348.93 =   347.71s
[junit4:junit4] JVM J6:     0.97 ..   130.46 =   129.49s
[junit4:junit4] JVM J7:     0.97 ..   118.79 =   117.82s
[junit4:junit4] Execution time total: 5 minutes 48 seconds
[junit4:junit4] Tests summary: 233 suites, 965 tests, 3 suite-level errors, 1 failure, 7 ignored (2 assumptions)

BUILD FAILED
<a class="moz-txt-link-rfc2396E" href="http://sierranevada.servebeer.com/job/fast-io-beasting/ws/build.xml">&lt;http://sierranevada.servebeer.com/job/fast-io-beasting/ws/build.xml&gt;</a>:39: The following error occurred while executing this line:
<a class="moz-txt-link-rfc2396E" href="http://sierranevada.servebeer.com/job/fast-io-beasting/ws/solr/build.xml">&lt;http://sierranevada.servebeer.com/job/fast-io-beasting/ws/solr/build.xml&gt;</a>:178: The following error occurred while executing this line:
<a class="moz-txt-link-rfc2396E" href="http://sierranevada.servebeer.com/job/fast-io-beasting/ws/lucene/module-build.xml">&lt;http://sierranevada.servebeer.com/job/fast-io-beasting/ws/lucene/module-build.xml&gt;</a>:61: The following error occurred while executing this line:
<a class="moz-txt-link-rfc2396E" href="http://sierranevada.servebeer.com/job/fast-io-beasting/ws/lucene/common-build.xml">&lt;http://sierranevada.servebeer.com/job/fast-io-beasting/ws/lucene/common-build.xml&gt;</a>:1159: The following error occurred while executing this line:
<a class="moz-txt-link-rfc2396E" href="http://sierranevada.servebeer.com/job/fast-io-beasting/ws/lucene/common-build.xml">&lt;http://sierranevada.servebeer.com/job/fast-io-beasting/ws/lucene/common-build.xml&gt;</a>:827: There were test failures: 233 suites, 965 tests, 3 suite-level errors, 1 failure, 7 ignored (2 assumptions)

Total time: 10 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure

---------------------------------------------------------------------
To unsubscribe, e-mail: <a class="moz-txt-link-abbreviated" href="mailto:dev-unsubscribe@lucene.apache.org">dev-unsubscribe@lucene.apache.org</a>
For additional commands, e-mail: <a class="moz-txt-link-abbreviated" href="mailto:dev-help@lucene.apache.org">dev-help@lucene.apache.org</a>

</pre>
    </blockquote>
    <br>
  </body>
</html>

--------------030108000501080704080901--