arrow-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Polchlopek <jpolchlo...@azavea.com>
Subject [Java] Increase memory limit
Date Thu, 06 Feb 2020 17:56:06 GMT
Hi,

I've been using Arrow Vectors as the underlying storage of some objects to
assist in the transmission of those objects, but as I am scaling up my
examples, I'm very quickly running into OOM errors.  Specifically

Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to
allocate 16777216 byte(s) of direct memory (used: 939524096, max:
954728448)

I'm a bit surprised.  The allocator managing the vectors is given by

val allocator = new RootAllocator(Long.MaxValue)

(I'm using the Java libs via Scala.)  I would expect to have more than
1GB available given that construction, but it looks like we're
substantially memory-constrained.

What can I do to increase the available pool of memory?

Thanks

-Justin

Mime
View raw message