From user-return-1114-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Wed Mar 24 03:29:25 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 431541804BB for ; Wed, 24 Mar 2021 04:29:25 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 5FFCE43834 for ; Wed, 24 Mar 2021 03:29:24 +0000 (UTC) Received: (qmail 15892 invoked by uid 500); 24 Mar 2021 03:29:23 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 15879 invoked by uid 99); 24 Mar 2021 03:29:23 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 24 Mar 2021 03:29:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id BFCABC02D5 for ; Wed, 24 Mar 2021 03:29:21 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: 0.199 X-Spam-Level: X-Spam-Status: No, score=0.199 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=airmettle-com.20150623.gappssmtp.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id IaQ_PQqGwEDY for ; Wed, 24 Mar 2021 03:29:21 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::32d; helo=mail-ot1-x32d.google.com; envelope-from=matt.youill@airmettle.com; receiver= Received: from mail-ot1-x32d.google.com (mail-ot1-x32d.google.com [IPv6:2607:f8b0:4864:20::32d]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id BCB3A7FC65 for ; Wed, 24 Mar 2021 03:29:20 +0000 (UTC) Received: by mail-ot1-x32d.google.com with SMTP id w21-20020a9d63950000b02901ce7b8c45b4so21683130otk.5 for ; Tue, 23 Mar 2021 20:29:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=airmettle-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=Kxr5KwhRQWxdoJxoyGWAihi6qy3uNijnQ4MOnLJJP6Y=; b=QAVsXQjGBBngxUvB4LWceb6Gf7FtTXnehWhvO4S7qZfm/htdpsnfayshSt5y6oaj+I ia37AY42dFqoW7ZjGxQ8Uxvf6Kv1Hsw8UPKd65zr2/6WgkspPzCdtZVSDC1gA8aD+8E9 +aem8NPo9R8+0fE7GN8lWop0y4fx/8wuWxRkKqWBXo2qY8FGyXo0G5sPgQ+JSsyrIXE6 TFoxD9LD68MQCEGWPEOGOoMuTkLBiDFXlH79n9Saa5mkbYK9aKE+5MaYyZZ6B0IoPtGR VXnHMyPTC+bllPT9GiYfZequW+m1EqXvmNYz6eLcwZ71qcZykwemT1sR6b9tLN1C9Q9Q fsCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=Kxr5KwhRQWxdoJxoyGWAihi6qy3uNijnQ4MOnLJJP6Y=; b=A61nB8n8t9Kg4sn1RULEBTAmHIddBaVDY6x+EbvoisWpLGUWVCwe/s23H7T6Jj2ZuX 5C++hj5E4ftksP7XXhEAwjo94wvAI1u3+uvb9Yqvs/FRnIzZ4Wl/J6ZXTdMKDaBge6dV kZTQQFXcLUQcX2mn0/65Ur2C4DNMltleAKBiWHs1pmnhXByEHQSRJXwPtlEJuk2pl3uB yzTW8Mn7OcZgoLNHoFsfdj20xmphf5JBOfZ7s3xjHEO7igyklj5yWNspKtGZuPNWjkjJ gc7XHCnD/+9cfexNyU4CJHaQPbwpSjbptjVT1ZNjeIjoS4Q5PGFqZIbGUEWB3eJvXKOp ewqw== X-Gm-Message-State: AOAM532GQqqfQXGQmAF3kehUW/nlXLOqF4WLyJ3wagMJwtxvICIHQhmz pZfzAjLsSwIL8SDiLKZynxeIP7In8wlU3YnXfmJR1WVWIn4XDQ== X-Google-Smtp-Source: ABdhPJyLW/mcK5OtNJbq/p64DsPwwV5u5iffUX5k4Axa2sEvauzwxugJ3Is/ZjbaPdbStC0skJ4oHC2MFUnezml59SU= X-Received: by 2002:a9d:6013:: with SMTP id h19mr1346532otj.72.1616556552822; Tue, 23 Mar 2021 20:29:12 -0700 (PDT) MIME-Version: 1.0 References: <20210322113538.17442986@fsol> <219f4c59-34a9-a273-2bcc-e587440b2910@airmettle.com> In-Reply-To: From: Matt Youill Date: Wed, 24 Mar 2021 14:29:00 +1100 Message-ID: Subject: Re: Cannot create default memory pool To: user@arrow.apache.org, emkornfield@gmail.com Content-Type: multipart/alternative; boundary="0000000000001733f705be3fe70e" --0000000000001733f705be3fe70e Content-Type: text/plain; charset="UTF-8" No CSV in these instances. On Wed., 24 Mar. 2021, 2:26 pm Micah Kornfield, wrote: > What is the source of the record batch? There was a patch since 3.0 that > fixed some potential memory corruption when reading parquet in certain > scenarios (but from the description it doesn't sound like libparquet is > being used?) > > On Tue, Mar 23, 2021 at 8:04 PM Matt Youill > wrote: > >> So this seems to be caused by the variable in memory_pool.cc: >> >> const util::optional user_selected_backend = >> UserSelectedBackend(); >> >> being (or becoming) garbage. >> >> For some reason, after a few Gandiva batch evaluations >> user_selected_backend is no longer "jemalloc" but "system" (probably >> actually just null because "system" is 0) and after a while it isn't valid >> at all and crashes. >> >> There aren't multiple copies of Arrow AFAICT but I do have two apps using >> arrow. Both use libarrow.a, libarrow-glib.a and libgandiva.a... one (that >> I'm not super familiar with) shows the above behavior and the other doesn't. >> >> On 22/3/21 10:27 pm, Matt Youill wrote: >> >> Could be the build creating multiple Arrows I suppose. It's a mixture of >> quite an old Makefile calling cmake to build arrow and arrow c lib. >> >> Will double check. >> >> Thanks, Matt >> >> On Mon., 22 Mar. 2021, 9:35 pm Antoine Pitrou, >> wrote: >> >>> On Mon, 22 Mar 2021 19:34:19 +1100 >>> Matt Youill wrote: >>> > Hi, >>> > >>> > Not sure if anyone knows anything about this, but am getting a strange >>> > error when evaluating a record batch with a gandiva filter... >>> > >>> > __GI_raise 0x00007f2b8f01718b >>> > __GI_abort 0x00007f2b8eff6859 >>> > arrow::util::ArrowLog::~ArrowLog() 0x000056309fe94c12 >>> > arrow::default_memory_pool() 0x000056309fd6fff4 >>> > gandiva::Annotator::PrepareEvalBatch(arrow::RecordBatch const&, >>> > std::vector, >>> > std::allocator > > const&) >>> > 0x000056309facdfce >>> > gandiva::LLVMGenerator::Execute(arrow::RecordBatch const&, >>> > std::vector, >>> > std::allocator > > const&) >>> > 0x000056309faa66a2 >>> > gandiva::Filter::Evaluate(arrow::RecordBatch const&, >>> > std::shared_ptr) 0x000056309fa9ea1d >>> > >>> > >>> > The error reported is "Internal error: cannot create default memory >>> pool" >>> > >>> > I'm using jemalloc >>> > >>> > Not even really sure how a call to arrow::default_memory_pool() can >>> > fail? This is only occurring in a release build if that helps? >>> >>> This logically should not happen. How did you compile Arrow and >>> Gandiva? Do you have two versions of Arrow lying around perhaps? >>> >>> >>> >>> --0000000000001733f705be3fe70e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
No CSV in these instances.

On Wed., 24 Mar. 2021, 2:26 pm = Micah Kornfield, <emkornfield@g= mail.com> wrote:
What is the source of the record batch?=C2=A0 There was a patch since= 3.0 that fixed some potential memory corruption when reading parquet in ce= rtain scenarios (but from the description it doesn't sound like libparq= uet is being used?)=C2=A0

On Tue, Mar 23, 2021 at 8:04 PM Matt Youill <matt.youill@airmettle.com> wrote:
=20 =20 =20

So this seems to be caused by the variable in memory_pool.cc:

const util::optional<MemoryPoolBackend> user_selected_backend =3D UserSelectedBackend();

being (or becoming) garbage.

For some reason, after a few Gandiva batch evaluations user_selected_backend is no longer "jemalloc" but "sys= tem" (probably actually just null because "system" is 0) and aft= er a while it isn't valid at all and crashes.

There aren't multiple copies of Arrow AFAICT but I do have two apps using arrow. Both use libarrow.a, libarrow-glib.a and libgandiva.a... one (that I'm not super familiar with) shows the above behavior and the other doesn't.


On 22/3/21 10:27 pm, Matt Youill wrote:
=20
Could be the build creating multiple Arrows I suppose. It'= s a mixture of quite an old Makefile calling cmake to build arrow and arrow c lib.

Will double check.

Thanks, Matt

On Mon., 22 Mar. 2021, 9:35 pm Antoine Pitrou, <antoine@python.org> wrote:
On Mon, 22 Mar 2021 19:34:19 +1100
Matt Youill <matt.youill@airmettle.com> wrote:
> Hi,
>
> Not sure if anyone knows anything about this, but am getting a strange
> error when evaluating a record batch with a gandiva filter...
>
> __GI_raise 0x00007f2b8f01718b
> __GI_abort 0x00007f2b8eff6859
> arrow::util::ArrowLog::~ArrowLog() 0x000056309fe94c12 > arrow::default_memory_pool() 0x000056309fd6fff4
> gandiva::Annotator::PrepareEvalBatch(arrow::RecordBatch const&,
> std::vector<std::shared_ptr<arrow::ArrayData>,
> std::allocator<std::shared_ptr<arrow::ArrayData> > > const&)
> 0x000056309facdfce
> gandiva::LLVMGenerator::Execute(arrow::RecordBatch const&,
> std::vector<std::shared_ptr<arrow::ArrayData>,
> std::allocator<std::shared_ptr<arrow::ArrayData> > > const&)
> 0x000056309faa66a2
> gandiva::Filter::Evaluate(arrow::RecordBatch const&,
> std::shared_ptr<gandiva::SelectionVector>) 0x000056309fa9ea1d
>
>
> The error reported is "Internal error: cannot creat= e default memory pool"
>
> I'm using jemalloc
>
> Not even really sure how a call to arrow::default_memory_pool() can
> fail? This is only occurring in a release build if that helps?

This logically should not happen.=C2=A0 How did you compile Arrow and
Gandiva?=C2=A0 Do you have two versions of Arrow lying around perhaps?



--0000000000001733f705be3fe70e--