arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fangzhou (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ARROW-198) OutOfMemoryError for vector test case
Date Fri, 13 May 2016 00:21:12 GMT

     [ https://issues.apache.org/jira/browse/ARROW-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Fangzhou updated ARROW-198:
---------------------------
    Description: 
When executing vector component  TestValueVector test cases(testVariableVectorReallocation,testFixedVectorReallocation),exception
can be seen.

Exception:
java.lang.Exception: Unexpected exception, expected<org.apache.arrow.vector.util.OversizedAllocationException>
but was<org.apache.arrow.memory.OutOfMemoryException>

where the issue comes from:
public void testFixedVectorReallocation() {
....
 try {
      vector.allocateNew(expectedValueCapacity);   <-- where the exception comes from
....

analysis:

allocateNew will invoke classes from io.netty.buffer from memory component and finally invoke
netty third party package.I assume there is no issue when invoking  memory  part.Then I trace
that the issue is caused by oversizing value of initialCapacity setting when invoking AbstractByteBufAllocator
class in third party netty-buffer-4.0.27.final.jar. I create a test case which can reproduces
the issue. value of parameter initialCapacity and maxCapacity are obtained from debugging
procedure

public void testAllocator(){
                int initialCapacity=2147483644;
		int maxCapacity=2147483647;
		ByteBuf buf = UnpooledByteBufAllocator.DEFAULT.directBuffer(initialCapacity, maxCapacity);
}

And I checked the rooted affected class java.nio.Bits:

static void reserveMemory(long size, int cap) {
.....
if (totalCapacity + cap > maxMemory)
    throw new OutOfMemoryError("Direct buffer memory");
.....

maxMemory can only be set to 1836056576 at most,but cap(initialCapacity) was 2147483644 at
this moment.

  was:
When executing vector component  junit-test class TestValueVector,exception can be seen.

Exception:
java.lang.Exception: Unexpected exception, expected<org.apache.arrow.vector.util.OversizedAllocationException>
but was<org.apache.arrow.memory.OutOfMemoryException>

where the issue comes from:
public void testFixedVectorReallocation() {
....
 try {
      vector.allocateNew(expectedValueCapacity);   <-- where the exception comes from
....

analysis:

allocateNew will invoke classes from io.netty.buffer from memory component and finally invoke
netty third party package.I assume there is no issue when invoking  memory  part.Then I trace
that the issue is caused by oversizing value of initialCapacity setting when invoking AbstractByteBufAllocator
class in third party netty-buffer-4.0.27.final.jar. I create a test case which can reproduces
the issue. value of parameter initialCapacity and maxCapacity are obtained from debugging
procedure

public void testAllocator(){
                int initialCapacity=2147483644;
		int maxCapacity=2147483647;
		ByteBuf buf = UnpooledByteBufAllocator.DEFAULT.directBuffer(initialCapacity, maxCapacity);
}

And I checked the rooted affected class java.nio.Bits:

static void reserveMemory(long size, int cap) {
.....
if (totalCapacity + cap > maxMemory)
    throw new OutOfMemoryError("Direct buffer memory");
.....

maxMemory can only be set to 1836056576 at most,but cap(initialCapacity) was 2147483644 at
this moment.


> OutOfMemoryError for vector test case
> -------------------------------------
>
>                 Key: ARROW-198
>                 URL: https://issues.apache.org/jira/browse/ARROW-198
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Java - Memory, Java - Vectors
>            Reporter: Fangzhou
>
> When executing vector component  TestValueVector test cases(testVariableVectorReallocation,testFixedVectorReallocation),exception
can be seen.
> Exception:
> java.lang.Exception: Unexpected exception, expected<org.apache.arrow.vector.util.OversizedAllocationException>
but was<org.apache.arrow.memory.OutOfMemoryException>
> where the issue comes from:
> public void testFixedVectorReallocation() {
> ....
>  try {
>       vector.allocateNew(expectedValueCapacity);   <-- where the exception comes from
> ....
> analysis:
> allocateNew will invoke classes from io.netty.buffer from memory component and finally
invoke netty third party package.I assume there is no issue when invoking  memory  part.Then
I trace that the issue is caused by oversizing value of initialCapacity setting when invoking
AbstractByteBufAllocator class in third party netty-buffer-4.0.27.final.jar. I create a test
case which can reproduces the issue. value of parameter initialCapacity and maxCapacity are
obtained from debugging procedure
> public void testAllocator(){
>                 int initialCapacity=2147483644;
> 		int maxCapacity=2147483647;
> 		ByteBuf buf = UnpooledByteBufAllocator.DEFAULT.directBuffer(initialCapacity, maxCapacity);
> }
> And I checked the rooted affected class java.nio.Bits:
> static void reserveMemory(long size, int cap) {
> .....
> if (totalCapacity + cap > maxMemory)
>     throw new OutOfMemoryError("Direct buffer memory");
> .....
> maxMemory can only be set to 1836056576 at most,but cap(initialCapacity) was 2147483644
at this moment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message