From user-return-317-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Thu Feb 6 19:35:11 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id F0A4918064E for ; Thu, 6 Feb 2020 20:35:10 +0100 (CET) Received: (qmail 70808 invoked by uid 500); 6 Feb 2020 19:35:10 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 70792 invoked by uid 99); 6 Feb 2020 19:35:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Feb 2020 19:35:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 7A4D41A31DC for ; Thu, 6 Feb 2020 19:35:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.001 X-Spam-Level: X-Spam-Status: No, score=0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=azavea.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id V_Qz9M9Mc5TS for ; Thu, 6 Feb 2020 19:35:07 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::22a; helo=mail-oi1-x22a.google.com; envelope-from=jpolchlopek@azavea.com; receiver= Received: from mail-oi1-x22a.google.com (mail-oi1-x22a.google.com [IPv6:2607:f8b0:4864:20::22a]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 4AD2A7DD70 for ; Thu, 6 Feb 2020 19:35:07 +0000 (UTC) Received: by mail-oi1-x22a.google.com with SMTP id q84so5824733oic.4 for ; Thu, 06 Feb 2020 11:35:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=azavea.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=Y9Do7Mh5jY9gG0ODFWg5m5hyvgGobxS/s8KW/fs6EJ8=; b=eTN0B656/z45siFFlaSiiPs0yIWUh2Jajz1lLIKPxzQFXI5ns/QQDO2qcf2hhuNLmR KLlYcyrEZx4lmAmRcGJhNVM+riJlyIv9GGunsVnaqwWFSTUT0J+SquWTF9du5BK88z0d 8BqsHB79djmOnUzgaCuAzg2YwkyHF97CMRwPc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=Y9Do7Mh5jY9gG0ODFWg5m5hyvgGobxS/s8KW/fs6EJ8=; b=HLnigYF7ivd8drp7jmQiksKSJ8HqNGNq2LB3IG7+CGdGxwJhrh54n3bxsFLA36aO6B RY0DTkEYpi6cxprWC/izsnfJXsGEr9dquznIVu1wy9RU0F3ruxFUzNCF1/J165voeprQ X8kkCwwtBxIZaNXbrYAdL2R49CtJhy84FONj6VR5SDOglqu8PHjKeQpHhfdWCevaWUMy tdGeHTTb1Cz2FiBPhHwc1qpO0azi2Dy1RrJ9h5hA4HUPMtR2VR0QzC4RTp8UJRZ93Fz6 UzGrVWsJ7C7U2JiaiW68vtlB1oigA6Be0InaItjOoJPHDDuvojfMywP0NuFqWeDdhg2S Mz9Q== X-Gm-Message-State: APjAAAW0hEl/nZSv6/NRLChRLU4PvpYIc3RXe1I43F+21Z3W8gmu+Zwh fVxNfCaleTfWOasEJGsTDizWHqDMVIH1twz2cnRfphYbFG8= X-Google-Smtp-Source: APXvYqxrpnhW3344O6wUVCgss6GF17AZ0H31deCD7LLj4jvT2o4K/K5yCxq7EVB4pNR0ciHUzk3e4y4HOKDn9/+ZuY4= X-Received: by 2002:aca:4ad8:: with SMTP id x207mr7689642oia.55.1581017705864; Thu, 06 Feb 2020 11:35:05 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Justin Polchlopek Date: Thu, 6 Feb 2020 14:34:55 -0500 Message-ID: Subject: Re: [Java] Increase memory limit To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000bdf0f4059ded5edd" --000000000000bdf0f4059ded5edd Content-Type: text/plain; charset="UTF-8" Great. That works. I should have checked something like this first. Thanks. On Thu, Feb 6, 2020 at 1:34 PM Jacques Nadeau wrote: > The jvm imposes a limit as well. The value you pass into the allocators > only matters if it is below the jvm setting. You'll need to change the jvm > setting via -XX:MaxDirectMemorySize= > > https://www.eclipse.org/openj9/docs/xxmaxdirectmemorysize/ > > On Thu, Feb 6, 2020 at 9:56 AM Justin Polchlopek > wrote: > >> Hi, >> >> I've been using Arrow Vectors as the underlying storage of some objects >> to assist in the transmission of those objects, but as I am scaling up my >> examples, I'm very quickly running into OOM errors. Specifically >> >> Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 939524096, max: 954728448) >> >> I'm a bit surprised. The allocator managing the vectors is given by >> >> val allocator = new RootAllocator(Long.MaxValue) >> >> (I'm using the Java libs via Scala.) I would expect to have more than 1GB available given that construction, but it looks like we're substantially memory-constrained. >> >> What can I do to increase the available pool of memory? >> >> Thanks >> >> -Justin >> >> --000000000000bdf0f4059ded5edd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Great.=C2=A0 That works.=C2=A0 I should have checked = something like this first.=C2=A0 Thanks.

On Thu, Feb 6, 2020 at 1:= 34 PM Jacques Nadeau <jacques@apac= he.org> wrote:
The jvm imposes a limit as well. The value you pass = into the allocators only matters if it is below the jvm setting. You'll= need to change the jvm setting via -XX:MaxDirectMemorySize=3D<size><= div>

On Thu, Feb 6, 2020 at 9:56 AM Justin Polchlo= pek <jpolchl= opek@azavea.com> wrote:
Hi,

I've = been using Arrow Vectors as the underlying storage of some objects to assis= t in the transmission of those objects, but as I am scaling up my examples,= I'm very quickly running into OOM errors.=C2=A0 Specifically 

Caused by: io.netty.util.internal.OutOfDirectMemo=
ryError: failed to allocate 16777216 byte(s) of direct memory (used: 939524=
096, max: 954728448)

I'm a bit =
surprised.  The allocator managing the vectors is given by

va=
l allocator =3D new RootAllocator(Long.MaxValue)
(I'm using the Java libs via Scala.)  I would exp=
ect to have more than 1GB available given that construction, but it looks l=
ike we're substantially memory-constrained.

What can I do to increase the available pool =
of memory?

Thanks<=
br>
-Justin
--000000000000bdf0f4059ded5edd--