mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <gsing...@apache.org>
Subject Re: Random Errors
Date Sun, 09 Jun 2013 22:03:58 GMT
Here's another theory:

I've noticed in my own usage of Hadoop that using the ToolRunner.run method as part of unit
tests often leads to bad things (it doesn't always exit cleanly, it seems) and we use it in
a lot of places.  Anyone else experience this?


On Jun 9, 2013, at 5:53 PM, Grant Ingersoll <gsingers@apache.org> wrote:

> Another one:
> 
> Tests run: 100, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 3.266 sec <<<
FAILURE!
> testViewSequentialAccessSparseVectorWritable {#18 seed=[D60713E4B2CC78FF:9F0DB2A85C2C8731]}(org.apache.mahout.math.VectorWritableTest)
 Time elapsed: 0.128 sec  <<< ERROR!
> com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from TEST scope at
testViewSequentialAccessSparseVectorWritable {#18 seed=[D60713E4B2CC78FF:9F0DB2A85C2C8731]}(org.apache.mahout.math.VectorWritableTest):

>   1) Thread[id=13, name=Thread-2, state=TERMINATED, group={null group}]
>        at (empty stack)
> 	at __randomizedtesting.SeedInfo.seed([D60713E4B2CC78FF:9F0DB2A85C2C8731]:0)
> 
> 
> On Jun 9, 2013, at 10:59 AM, Dawid Weiss <dawid.weiss@cs.put.poznan.pl> wrote:
> 
>>> com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from TEST
scope at
>> 
>> This is because the test suite extends RandomizedTest and this in turn
>> enables a feature of the runner named aptly "thread leak detection".
>> The problem is that if you spawn threads from a test and then return
>> to the framework those threads running in the background may affect
>> further computations and be very difficult to debug. The runner
>> detects such cases and fails a test that didn't clean up after itself
>> properly.
>> 
>> There are a few workarounds:
>> 
>> 1) do clean up; after the test is over all threads it possibly starts
>> should be join()'ed with.
>> 
>> 2) sometimes the above is not possible explicitly -- as with
>> executors, for example. If it's known that threads may linger a bit
>> but will eventually terminate a @ThreadLeakLingering(linger = 20000)
>> can be applied which gives the maximum time to wait for stray threads.
>> They still must terminate.
>> 
>> 3) sometimes threads should last for the entire suite's duration. The
>> scope of detection can be changed with @ThreadLeakScope(Scope.SUITE).
>> 
>> 4) finally, if this all is not really needed the feature can be
>> disabled by @ThreadLeakScope(Scope.NONE). I honestly think this is the
>> worst scenario though because leaked threads are *very* difficult to
>> debug if they do something wrong and affect test results.
>> 
>> Take a look at class annotations of:
>> http://goo.gl/n7rYD
>> 
>> for an example of multiple configuration directives used in real life:
>> 
>> @ThreadLeakScope(Scope.SUITE)
>> @ThreadLeakGroup(Group.MAIN)
>> @ThreadLeakAction({Action.WARN, Action.INTERRUPT})
>> @ThreadLeakLingering(linger = 20000) // Wait long for leaked threads
>> to complete before failure. zk needs this.
>> @ThreadLeakZombies(Consequence.IGNORE_REMAINING_TESTS)
>> @TimeoutSuite(millis = 2 * TimeUnits.HOUR)
>> @ThreadLeakFilters(defaultFilters = true, filters = {
>>    QuickPatchThreadsFilter.class
>> })
>> 
>> Dawid
> 
> --------------------------------------------
> Grant Ingersoll | @gsingers
> http://www.lucidworks.com
> 
> 
> 
> 
> 

--------------------------------------------
Grant Ingersoll | @gsingers
http://www.lucidworks.com






Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message