arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Polchlopek <jpolchlo...@azavea.com>
Subject Re: [Java] Increase memory limit
Date Thu, 06 Feb 2020 19:34:55 GMT
Great.  That works.  I should have checked something like this first.
Thanks.

On Thu, Feb 6, 2020 at 1:34 PM Jacques Nadeau <jacques@apache.org> wrote:

> The jvm imposes a limit as well. The value you pass into the allocators
> only matters if it is below the jvm setting. You'll need to change the jvm
> setting via -XX:MaxDirectMemorySize=<size>
>
> https://www.eclipse.org/openj9/docs/xxmaxdirectmemorysize/
>
> On Thu, Feb 6, 2020 at 9:56 AM Justin Polchlopek <jpolchlopek@azavea.com>
> wrote:
>
>> Hi,
>>
>> I've been using Arrow Vectors as the underlying storage of some objects
>> to assist in the transmission of those objects, but as I am scaling up my
>> examples, I'm very quickly running into OOM errors.  Specifically
>>
>> Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216
byte(s) of direct memory (used: 939524096, max: 954728448)
>>
>> I'm a bit surprised.  The allocator managing the vectors is given by
>>
>> val allocator = new RootAllocator(Long.MaxValue)
>>
>> (I'm using the Java libs via Scala.)  I would expect to have more than 1GB available
given that construction, but it looks like we're substantially memory-constrained.
>>
>> What can I do to increase the available pool of memory?
>>
>> Thanks
>>
>> -Justin
>>
>>

Mime
View raw message