Great, good luck on your use case!

On Thu, Feb 6, 2020 at 11:35 AM Justin Polchlopek <jpolchlopek@azavea.com> wrote:
Great.  That works.  I should have checked something like this first.  Thanks.

On Thu, Feb 6, 2020 at 1:34 PM Jacques Nadeau <jacques@apache.org> wrote:
The jvm imposes a limit as well. The value you pass into the allocators only matters if it is below the jvm setting. You'll need to change the jvm setting via -XX:MaxDirectMemorySize=<size>

On Thu, Feb 6, 2020 at 9:56 AM Justin Polchlopek <jpolchlopek@azavea.com> wrote:
Hi,

I've been using Arrow Vectors as the underlying storage of some objects to assist in the transmission of those objects, but as I am scaling up my examples, I'm very quickly running into OOM errors.  Specifically

Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 939524096, max: 954728448)

I'm a bit surprised.  The allocator managing the vectors is given by

val allocator = new RootAllocator(Long.MaxValue)
(I'm using the Java libs via Scala.)  I would expect to have more than 1GB available given that construction, but it looks like we're substantially memory-constrained.

What can I do to increase the available pool of memory?

Thanks
-Justin