hbase-builds mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Build failed in Jenkins: HBase-TRUNK #5502
Date Sat, 13 Sep 2014 17:39:51 GMT
This one is odd:  java.lang.OutOfMemoryError: unable to create new native
thread

It happens in the run of small tests. I've not see this before. If it
happens again I'll try upping the jstack sizes when tests run.

St.Ack

On Fri, Sep 12, 2014 at 10:08 PM, Apache Jenkins Server <
jenkins@builds.apache.org> wrote:

> See <https://builds.apache.org/job/HBase-TRUNK/5502/changes>
>
> Changes:
>
> [stack] HBASE-11911 Break up tests into more fine grained categories (Alex
> Newman)
>
> ------------------------------------------
> [...truncated 331 lines...]
> Running org.apache.hadoop.hbase.TestKeyValue
> Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec
> - in org.apache.hadoop.hbase.TestKeyValue
> Running org.apache.hadoop.hbase.codec.TestCellCodecWithTags
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.151 sec
> - in org.apache.hadoop.hbase.codec.TestCellCodecWithTags
> Running org.apache.hadoop.hbase.codec.TestKeyValueCodecWithTags
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.179 sec
> - in org.apache.hadoop.hbase.codec.TestKeyValueCodecWithTags
> Running org.apache.hadoop.hbase.codec.TestCellCodec
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.169 sec
> - in org.apache.hadoop.hbase.codec.TestCellCodec
> Running org.apache.hadoop.hbase.codec.TestKeyValueCodec
> Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 sec
> - in org.apache.hadoop.hbase.codec.TestKeyValueCodec
> Running org.apache.hadoop.hbase.TestClassFinder
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.574 sec
> - in org.apache.hadoop.hbase.TestClassFinder
> Running org.apache.hadoop.hbase.io.TestTagCompressionContext
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.028 sec
> <<< FAILURE! - in org.apache.hadoop.hbase.io.TestTagCompressionContext
> org.apache.hadoop.hbase.io.TestTagCompressionContext  Time elapsed: 0.027
> sec  <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:693)
>         at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>         at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)
>         at java.lang.UNIXProcess.initStreams(UNIXProcess.java:172)
>         at java.lang.UNIXProcess$2.run(UNIXProcess.java:145)
>         at java.lang.UNIXProcess$2.run(UNIXProcess.java:143)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.lang.UNIXProcess.<init>(UNIXProcess.java:143)
>         at java.lang.ProcessImpl.start(ProcessImpl.java:130)
>         at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)
>         at java.lang.Runtime.exec(Runtime.java:617)
>         at java.lang.Runtime.exec(Runtime.java:450)
>         at java.lang.Runtime.exec(Runtime.java:347)
>         at
> org.apache.hadoop.hbase.util.JVM.getNumberOfRunningProcess(JVM.java:198)
>         at
> org.apache.hadoop.hbase.ResourceCheckerJUnitListener$ProcessCountResourceAnalyzer.getVal(ResourceCheckerJUnitListener.java:123)
>         at
> org.apache.hadoop.hbase.ResourceChecker.fill(ResourceChecker.java:114)
>         at
> org.apache.hadoop.hbase.ResourceChecker.fillInit(ResourceChecker.java:103)
>         at
> org.apache.hadoop.hbase.ResourceChecker.start(ResourceChecker.java:186)
>         at
> org.apache.hadoop.hbase.ResourceCheckerJUnitListener.start(ResourceCheckerJUnitListener.java:156)
>         at
> org.apache.hadoop.hbase.ResourceCheckerJUnitListener.testStarted(ResourceCheckerJUnitListener.java:179)
>         at
> org.junit.runner.notification.RunNotifier$3.notifyListener(RunNotifier.java:115)
>         at
> org.junit.runner.notification.RunNotifier$SafeNotifier.run(RunNotifier.java:61)
>         at
> org.junit.runner.notification.RunNotifier.fireTestStarted(RunNotifier.java:112)
>         at
> org.junit.internal.runners.model.EachTestNotifier.fireTestStarted(EachTestNotifier.java:43)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:269)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>         at org.junit.runners.Suite.runChild(Suite.java:127)
>         at org.junit.runners.Suite.runChild(Suite.java:26)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:113)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:85)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:134)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>
> Running org.apache.hadoop.hbase.io.util.TestLRUDictionary
> Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.357 sec
> - in org.apache.hadoop.hbase.io.util.TestLRUDictionary
> Running org.apache.hadoop.hbase.io.crypto.TestKeyStoreKeyProvider
> Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.029 sec
> <<< FAILURE! - in org.apache.hadoop.hbase.io.crypto.TestKeyStoreKeyProvider
> testKeyStoreKeyProviderWithPasswordFile(org.apache.hadoop.hbase.io.crypto.TestKeyStoreKeyProvider)
> Time elapsed: 0.024 sec  <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:693)
>         at
> org.junit.internal.runners.statements.FailOnTimeout.evaluateStatement(FailOnTimeout.java:25)
>         at
> org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:17)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>         at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>         at org.junit.runners.Suite.runChild(Suite.java:127)
>         at org.junit.runners.Suite.runChild(Suite.java:26)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:113)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:85)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:134)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>
> testKeyStoreKeyProviderWithPassword(org.apache.hadoop.hbase.io.crypto.TestKeyStoreKeyProvider)
> Time elapsed: 0.002 sec  <<< ERROR!
> java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method)
>         at java.lang.Thread.start(Thread.java:693)
>         at
> org.junit.internal.runners.statements.FailOnTimeout.evaluateStatement(FailOnTimeout.java:25)
>         at
> org.junit.internal.runners.statements.FailOnTimeout.evaluate(FailOnTimeout.java:17)
>         at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>         at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>         at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>         at org.junit.runners.Suite.runChild(Suite.java:127)
>         at org.junit.runners.Suite.runChild(Suite.java:26)
>         at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>         at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>         at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>         at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>         at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>         at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
>         at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:113)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:85)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
>         at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:134)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
>         at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
>
> Running org.apache.hadoop.hbase.io.crypto.aes.TestAES
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.784 sec
> - in org.apache.hadoop.hbase.io.crypto.aes.TestAES
> Running org.apache.hadoop.hbase.io.crypto.TestEncryption
> Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.834 sec
> - in org.apache.hadoop.hbase.io.crypto.TestEncryption
> Running org.apache.hadoop.hbase.io.crypto.TestCipherProvider
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.647 sec
> - in org.apache.hadoop.hbase.io.crypto.TestCipherProvider
> Running org.apache.hadoop.hbase.io.crypto.TestKeyProvider
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.344 sec
> - in org.apache.hadoop.hbase.io.crypto.TestKeyProvider
> Running org.apache.hadoop.hbase.TestCompoundConfiguration
> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.423 sec
> - in org.apache.hadoop.hbase.TestCompoundConfiguration
> Running org.apache.hadoop.hbase.TestHBaseConfiguration
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.379 sec
> - in org.apache.hadoop.hbase.TestHBaseConfiguration
>
> Results :
>
> Tests in error:
>   TestCounter.testIncrement:41->execute:94 ? OutOfMemory unable to create
> new na...
>   TestCounter.testIncrementAndGet:57->execute:94 ? OutOfMemory unable to
> create ...
>   TestDrainBarrier.testStopIsBlockedByOps:92 ? OutOfMemory unable to
> create new ...
>
> TestCoprocessorClassLoader.testRelativeLibJarName:78->checkingLibJarName:101
> ? IO
>   TestCoprocessorClassLoader.testCleanupOldJars:66 ? IO Cannot run program
> "chmo...
>   TestCoprocessorClassLoader.testLibJarName:73->checkingLibJarName:101 ?
> IO Cann...
>   TestThreads.testSleepWithoutInterrupt ? OutOfMemory unable to create new
> nativ...
>
> TestTagCompressionContext.org.apache.hadoop.hbase.io.TestTagCompressionContext
> ? OutOfMemory
>   TestKeyStoreKeyProvider.testKeyStoreKeyProviderWithPasswordFile ?
> OutOfMemory ...
>   TestKeyStoreKeyProvider.testKeyStoreKeyProviderWithPassword ?
> OutOfMemory unab...
>
> Tests run: 160, Failures: 0, Errors: 10, Skipped: 0
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] HBase ............................................. SUCCESS [2.491s]
> [INFO] HBase - Common .................................... FAILURE
> [37.973s]
> [INFO] HBase - Protocol .................................. SKIPPED
> [INFO] HBase - Client .................................... SKIPPED
> [INFO] HBase - Hadoop Compatibility ...................... SKIPPED
> [INFO] HBase - Hadoop Two Compatibility .................. SKIPPED
> [INFO] HBase - Prefix Tree ............................... SKIPPED
> [INFO] HBase - Server .................................... SKIPPED
> [INFO] HBase - Testing Util .............................. SKIPPED
> [INFO] HBase - Thrift .................................... SKIPPED
> [INFO] HBase - Shell ..................................... SKIPPED
> [INFO] HBase - Integration Tests ......................... SKIPPED
> [INFO] HBase - Examples .................................. SKIPPED
> [INFO] HBase - Assembly .................................. SKIPPED
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 41.334s
> [INFO] Finished at: Sat Sep 13 05:06:52 UTC 2014
> [INFO] Final Memory: 37M/413M
> [INFO]
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on
> project hbase-common: ExecutionException:
> org.apache.maven.surefire.booter.SurefireBooterForkException: Error
> occurred in starting fork, check output in log -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn <goals> -rf :hbase-common
> Build step 'Invoke top-level Maven targets' marked build as failure
> Performing Post build task...
> Match found for :.* : True
> Logical operation result is TRUE
> Running script  :   ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l`
>   if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
>     #It seems sometimes the tests are not dying immediately. Let's give
> them 10s
>     echo "Suspicious java process found - waiting 10s to see if there are
> just slow to stop"
>     sleep 10
>     ZOMBIE_TESTS_COUNT=`jps | grep surefirebooter | wc -l`
>     if [[ $ZOMBIE_TESTS_COUNT != 0 ]] ; then
>       echo "There are $ZOMBIE_TESTS_COUNT zombie tests, they should have
> been killed by surefire but survived"
>       echo "************ BEGIN zombies jstack extract"
>       ZB_STACK=`jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1
> jstack | grep ".test" | grep "\.java"`
>       jps | grep surefirebooter | cut -d ' ' -f 1 | xargs -n 1 jstack
>       echo "************ END  zombies jstack extract"
>       JIRA_COMMENT="$JIRA_COMMENT
>
>      {color:red}-1 core zombie tests{color}.  There are
> ${ZOMBIE_TESTS_COUNT} zombie test(s): ${ZB_STACK}"
>       BAD=1
>       jps | grep surefirebooter | cut -d ' ' -f 1 | xargs kill -9
>     else
>       echo "We're ok: there is no zombie test, but some tests took some
> time to stop"
>     fi
>   else
>     echo "We're ok: there is no zombie test"
>   fi
> [HBase-TRUNK] $ /bin/bash -xe /tmp/hudson2064141739698639434.sh
> ++ jps
> ++ grep surefirebooter
> ++ wc -l
> + ZOMBIE_TESTS_COUNT=0
> + [[ 0 != 0 ]]
> + echo 'We'\''re ok: there is no zombie test'
> We're ok: there is no zombie test
> POST BUILD TASK : SUCCESS
> END OF POST BUILD TASK : 0
> Archiving artifacts
> Sending artifact delta relative to HBase-TRUNK #5500
> Archived 128 artifacts
> Archive block size is 32768
> Received 8 blocks and 538407 bytes
> Compression is 32.7%
> Took 50 sec
> Recording test results
> Updating HBASE-11911
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message