From user-return-318-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Thu Feb 6 19:45:11 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 7E42918064E for ; Thu, 6 Feb 2020 20:45:11 +0100 (CET) Received: (qmail 83965 invoked by uid 500); 6 Feb 2020 19:45:10 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 83953 invoked by uid 99); 6 Feb 2020 19:45:10 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Feb 2020 19:45:10 +0000 Received: from mail-ot1-f52.google.com (mail-ot1-f52.google.com [209.85.210.52]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 9673E2E49 for ; Thu, 6 Feb 2020 19:45:10 +0000 (UTC) Received: by mail-ot1-f52.google.com with SMTP id z9so6681798oth.5 for ; Thu, 06 Feb 2020 11:45:10 -0800 (PST) X-Gm-Message-State: APjAAAVLbqrFqMPPnIP96EQWskMDQOIULtISwlYlYGNuB3xuNb191ow0 kkwpnVTExJizWydMXqq8l0Cd+pslJv6nmokjCxA= X-Google-Smtp-Source: APXvYqyKpgJ9xtMWQ4+Dg/5isZ8RIVqmekjzDaVzS1L0Wavqjw0pUktnL/9AyzS9ZuCmshQhnc1vttk1oX676H9frTQ= X-Received: by 2002:a05:6830:15c5:: with SMTP id j5mr30759939otr.289.1581018309819; Thu, 06 Feb 2020 11:45:09 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Jacques Nadeau Date: Thu, 6 Feb 2020 11:44:58 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [Java] Increase memory limit To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000bd83b7059ded821a" --000000000000bd83b7059ded821a Content-Type: text/plain; charset="UTF-8" Great, good luck on your use case! On Thu, Feb 6, 2020 at 11:35 AM Justin Polchlopek wrote: > Great. That works. I should have checked something like this first. > Thanks. > > On Thu, Feb 6, 2020 at 1:34 PM Jacques Nadeau wrote: > >> The jvm imposes a limit as well. The value you pass into the allocators >> only matters if it is below the jvm setting. You'll need to change the jvm >> setting via -XX:MaxDirectMemorySize= >> >> https://www.eclipse.org/openj9/docs/xxmaxdirectmemorysize/ >> >> On Thu, Feb 6, 2020 at 9:56 AM Justin Polchlopek >> wrote: >> >>> Hi, >>> >>> I've been using Arrow Vectors as the underlying storage of some objects >>> to assist in the transmission of those objects, but as I am scaling up my >>> examples, I'm very quickly running into OOM errors. Specifically >>> >>> Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 939524096, max: 954728448) >>> >>> I'm a bit surprised. The allocator managing the vectors is given by >>> >>> val allocator = new RootAllocator(Long.MaxValue) >>> >>> (I'm using the Java libs via Scala.) I would expect to have more than 1GB available given that construction, but it looks like we're substantially memory-constrained. >>> >>> What can I do to increase the available pool of memory? >>> >>> Thanks >>> >>> -Justin >>> >>> --000000000000bd83b7059ded821a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Great, good luck on your use case!

On Thu, Feb 6, 2020 at 1= 1:35 AM Justin Polchlopek <jpo= lchlopek@azavea.com> wrote:
Great.=C2=A0 That works.=C2=A0 I s= hould have checked something like this first.=C2=A0 Thanks.
=
On Thu= , Feb 6, 2020 at 1:34 PM Jacques Nadeau <jacques@apache.org> wrote:
The jvm imposes= a limit as well. The value you pass into the allocators only matters if it= is below the jvm setting. You'll need to change the jvm setting via -X= X:MaxDirectMemorySize=3D<size>
On Thu, = Feb 6, 2020 at 9:56 AM Justin Polchlopek <jpolchlopek@azavea.com> wrote:
H= i,

I've been using Arrow Vectors as the underl= ying storage of some objects to assist in the transmission of those objects= , but as I am scaling up my examples, I'm very quickly running into OOM= errors.=C2=A0 Specifically 

Caused by: i=
o.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 b=
yte(s) of direct memory (used: 939524096, max: 954728448)

I'm a bit surprised.  The allocator managing the=
 vectors is given by

val allocator =3D new RootAllocator(Long=
.MaxValue)
(I'm using the=
 Java libs via Scala.)  I would expect to have more than 1GB available give=
n that construction, but it looks like we're substantially memory-const=
rained.

What can I=
 do to increase the available pool of memory?

Thanks
-Justin
--000000000000bd83b7059ded821a--