hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From László Bodor (Jira) <j...@apache.org>
Subject [jira] [Commented] (HIVE-22277) TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
Date Tue, 01 Oct 2019 09:20:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-22277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16941663#comment-16941663
] 

László Bodor commented on HIVE-22277:
-------------------------------------

I'm not familiar with this test case, but we should choose a way to go:
1. @RunWith retry logic (best effort, doesn't solve the original problem) on AllocatorOutOfMemoryException
2. review the resource needs for this job and change infra accordingly
3. disable test
what do you think [~bslim]?

> TestBuddyAllocator.testMTT[2] is flaky (direct=true, mmap=true)
> ---------------------------------------------------------------
>
>                 Key: HIVE-22277
>                 URL: https://issues.apache.org/jira/browse/HIVE-22277
>             Project: Hive
>          Issue Type: Bug
>            Reporter: László Bodor
>            Priority: Major
>         Attachments: hive.log.tar.gz
>
>
> It's more than flaky. Recently, in most of the cases it's the only failing test in otherwise
green runs.
> full hive logs:  [^hive.log.tar.gz] 
> {code}
> Error Message
> java.util.concurrent.ExecutionException: org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked, or an internal
issue)
> Stacktrace
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked, or an internal
issue)
> 	at org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:149)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> 	at org.junit.runners.Suite.runChild(Suite.java:127)
> 	at org.junit.runners.Suite.runChild(Suite.java:26)
> 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> 	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> 	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
> 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
> 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
> 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
> Caused by: java.util.concurrent.ExecutionException: org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked, or an internal
issue)
> 	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> 	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> 	at org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT(TestBuddyAllocator.java:145)
> 	... 33 more
> Caused by: org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
Failed to allocate 255; at 1 out of 3 (entire cache is fragmented and locked, or an internal
issue)
> 	at org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:454)
> 	at org.apache.hadoop.hive.llap.cache.BuddyAllocator.allocateMultiple(BuddyAllocator.java:299)
> 	at org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.allocateAndUseBuffer(TestBuddyAllocator.java:254)
> 	at org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.allocateUp(TestBuddyAllocator.java:231)
> 	at org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.access$000(TestBuddyAllocator.java:43)
> 	at org.apache.hadoop.hive.llap.cache.TestBuddyAllocator$1.call(TestBuddyAllocator.java:119)
> 	at org.apache.hadoop.hive.llap.cache.TestBuddyAllocator$1.call(TestBuddyAllocator.java:116)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:745)
> Standard Error
> 2019-09-30T03:04:43,170  INFO [main] LlapIoImpl: Buddy allocator with direct buffers;
memory mapped off /home/hiveptest/104.197.225.41-hiveptest-1/apache-github-source-source/llap-server/target/tmp/llap-5698481178354861395;
allocation sizes 8 - 256, arena size 256, total size 6144
> 2019-09-30T03:04:43,174  WARN [pool-6-thread-3] LlapIoImpl: Failed to allocate [3]X[256]
bytes after [15] attempt, evicted [0] bytes and partially allocated [256] bytes
> 2019-09-30T03:04:43,174  WARN [pool-6-thread-2] LlapIoImpl: Failed to allocate [3]X[256]
bytes after [15] attempt, evicted [0] bytes and partially allocated [0] bytes
> 2019-09-30T03:04:43,175 ERROR [pool-6-thread-2] LlapIoImpl: Failed to allocate 255; at
1 out of 3 (entire cache is fragmented and locked, or an internal issue)
> 2019-09-30T03:04:43,175 ERROR [pool-6-thread-3] LlapIoImpl: Failed to allocate 255; at
1 out of 3 (entire cache is fragmented and locked, or an internal issue)
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 
>   block 0 at 0: size 256, allocated
> Arena: 
>   free list for size 8: 5, 
>   free list for size 16: 0, 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 
>   block 0 at 0: size 16, free
>   block 2 at 16: size 8, allocated
>   block 3 at 24: size 8, allocated
>   block 4 at 32: size 8, allocated
>   block 5 at 40: size 8, free
>   block 6 at 48: size 16, allocated
>   block 8 at 64: size 64, allocated
>   block 16 at 128: size 128, allocated
> Arena: 
>   free list for size 8: 
>   free list for size 16: 6, 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 
>   block 0 at 0: size 16, free
>   block 2 at 16: size 16, allocated
>   block 4 at 32: size 16, allocated
>   block 6 at 48: size 16, free
>   block 8 at 64: size 64, allocated
>   block 16 at 128: size 128, allocated
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 0, 
>   free list for size 64: 
>   free list for size 128: 16, 
>   free list for size 256: 
>   block 0 at 0: size 32, free
>   block 4 at 32: size 32, allocated
>   block 8 at 64: size 32, allocated
>   block 12 at 96: size 32, allocated
>   block 16 at 128: size 128, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 
>   block 0 at 0: size 64, allocated
>   block 8 at 64: size 64, allocated
>   block 16 at 128: size 128, allocated
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 
>   block 0 at 0: size 64, allocated
>   block 8 at 64: size 64, allocated
>   block 16 at 128: size 64, allocated
>   block 24 at 192: size 64, allocated
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 8, 
>   free list for size 128: 16, 
>   free list for size 256: 
>   block 0 at 0: size 64, free
>   block 8 at 64: size 64, free
>   block 16 at 128: size 128, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> Arena: 
>   free list for size 8: 
>   free list for size 16: 
>   free list for size 32: 
>   free list for size 64: 
>   free list for size 128: 
>   free list for size 256: 0, 
>   block 0 at 0: size 256, free
> 2019-09-30T03:04:43,176 ERROR [pool-6-thread-2] cache.TestBuddyAllocator: Failed to allocate
3 of 255; {[256.@0]}, {[16.@0][8*@2][8*@3][8*@4][8.@5][16*@6][64*@8][128*@16]}, {[16.@0][16*@2][16*@4][16.@6][64*@8][128*@16]},
{[32.@0][32*@4][32*@8][32*@12][128.@16]}, {[64.@0][64*@8][128*@16]}, {[256.@0]}, {[256.@0]},
{[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]},
{[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]},
{[256.@0]}, 
> 2019-09-30T03:04:43,177 ERROR [pool-6-thread-3] cache.TestBuddyAllocator: Failed to allocate
3 of 255; {[256.@0]}, {[16.@0][8*@2][8*@3][8*@4][8.@5][16*@6][64*@8][128*@16]}, {[16.@0][16*@2][16*@4][16.@6][64*@8][128*@16]},
{[32.@0][32*@4][32*@8][32*@12][128.@16]}, {[64.@0][64*@8][128*@16]}, {[256.@0]}, {[256.@0]},
{[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]},
{[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]}, {[256.@0]},
{[256.@0]}, 
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message