Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Received-SPF: pass (nike.apache.org: message received from 54.76.25.247 which
 is an MX secondary for dev@drill.apache.org)
From: Sudheesh Katkam <skatkam@maprtech.com>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Subject: Re: TestDrillbitResilience broken? assertion errors;
 now slow/hung, with 278 threads!
Message-Id: <921C02F9-9733-4585-9290-AAE798797F67@maprtech.com>
Date: Wed, 29 Apr 2015 08:07:05 -0700
References: <55408487.7050106@maprtech.com>
 <CAKTYAC-2Px4=KoN0sOs9R+tqVYw_YcmyUjV4B56ZSsG=8mZEBw@mail.gmail.com>
 <269638FA-9B80-4F81-A0AA-A8C48B32BFDA@maprtech.com>
In-Reply-To: <269638FA-9B80-4F81-A0AA-A8C48B32BFDA@maprtech.com>
To: "dev@drill.apache.org" <dev@drill.apache.org>

*ran the tests before checking them in.=20

> On Apr 29, 2015, at 7:53 AM, Sudheesh Katkam <skatkam@maprtech.com> wrote:=

>=20
> I am responsible for those tests. I ran the tests at least 10 times on my L=
inux VM with 1 second pauses, all of which passed.=20
>=20
> On your second run, what different errors did you see?
>=20
> On your third run, are you able to reproduce the test case the hangs?
>=20
> Sorry that the message is not informative. I already have a patch which is=
 a slight improvement to Jacques change that improves the message in those t=
ests. =20
>=20
> What tool did you use to get the thread count?
>=20
> - Sudheesh
>=20
> Sent from my iPhone. Pardon any typos.
>=20
>> On Apr 29, 2015, at 6:28 AM, Abdel Hakim Deneche <adeneche@maprtech.com> w=
rote:
>>=20
>> The message displayed in the first run contains actually two different
>> issues:
>>=20
>> 1. The error message "Error shutting down Drillbit 'beta'" is most likely=

>> caused by this issue DRILL-2878
>> <https://issues.apache.org/jira/browse/DRILL-2878>
>>=20
>> 2. The test that failed with an "java.lang.AssertionError: null" is most
>> likely a bug because that unit test should not fail. I've seen this error=

>> before, but it only happens intermittently.
>>=20
>> The system error reported in the 3rd run is actually an "expected" inject=
ed
>> exception, but 278 threads looks suspicious!!!
>>=20
>> On Wed, Apr 29, 2015 at 12:13 AM, Daniel Barclay <dbarclay@maprtech.com>
>> wrote:
>>=20
>>> Does anyone know what's going on with TestDrillbitResilience (rebased
>>> from master today)?  (Is it working right?)
>>>=20
>>>=20
>>> One run, via "mvn install", yielded assertion errors:
>>>=20
>>> ...
>>> Error shutting down Drillbit "beta".
>>> Tests run: 11, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 33.811
>>> sec <<< FAILURE! - in org.apache.drill.exec.server.TestDrillbitResilienc=
e
>>> cancelAfterEverythingIsCompleted(org.apache.drill.exec.server.TestDrillb=
itResilience)
>>> Time elapsed: 1.468 sec  <<< FAILURE!
>>> java.lang.AssertionError: null
>>>       at
>>> org.apache.drill.exec.server.TestDrillbitResilience.assertCancelled(Test=
DrillbitResilience.java:459)
>>>       at
>>> org.apache.drill.exec.server.TestDrillbitResilience.cancelAfterEverythin=
gIsCompleted(TestDrillbitResilience.java:565)
>>>=20
>>> cancelInMiddleOfFetchingResults(org.apache.drill.exec.server.TestDrillbi=
tResilience)
>>> Time elapsed: 1.496 sec  <<< FAILURE!
>>> java.lang.AssertionError: null
>>>       at
>>> org.apache.drill.exec.server.TestDrillbitResilience.assertCancelled(Test=
DrillbitResilience.java:459)
>>>       at
>>> org.apache.drill.exec.server.TestDrillbitResilience.cancelInMiddleOfFetc=
hingResults(TestDrillbitResilience.java:510)
>>>=20
>>> Running <next test>
>>> ...
>>>=20
>>>=20
>>> A second run, run individually (but still via Maven) died with different=

>>> errors.
>>>=20
>>>=20
>>>=20
>>> A third run, via "mvn install" again, seems hung after reporting this
>>> (maybe expected) exception:
>>>=20
>>> Exception (no rows returned):
>>> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
>>> run-try-end
>>>=20
>>>=20
>>> [fb9cfe61-af6e-4c9c-b6ab-8a1b8725c6e9 on dev-linux2:31010]
>>>=20
>>>=20
>>> The process is using only about 5% CPU--but has 278 threads!
>>> (That includes about 35 threads all with the same name of "BitClient-1".=
)
>>>=20
>>>=20
>>> Daniel
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>> --
>>> Daniel Barclay
>>> MapR Technologies
>>=20
>>=20
>>=20
>> --=20
>>=20
>> Abdelhakim Deneche
>>=20
>> Software Engineer
>>=20
>> <http://www.mapr.com/>
>>=20
>>=20
>> Now Available - Free Hadoop On-Demand Training
>> <http://www.mapr.com/training?utm_source=3DEmail&utm_medium=3DSignature&u=
tm_campaign=3DFree%20available>