From user-return-705-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Mon Oct 12 15:10:22 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id C2F2218064E for ; Mon, 12 Oct 2020 17:10:22 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 5549664962 for ; Mon, 12 Oct 2020 15:10:06 +0000 (UTC) Received: (qmail 87408 invoked by uid 500); 12 Oct 2020 15:10:05 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 87398 invoked by uid 99); 12 Oct 2020 15:10:05 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 12 Oct 2020 15:10:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id BEFF1BFE15 for ; Mon, 12 Oct 2020 15:10:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=0 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id qNMMvaXBg_Gd for ; Mon, 12 Oct 2020 15:10:03 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2607:f8b0:4864:20::636; helo=mail-pl1-x636.google.com; envelope-from=neal.p.richardson@gmail.com; receiver= Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 343C27F9AD for ; Mon, 12 Oct 2020 15:10:03 +0000 (UTC) Received: by mail-pl1-x636.google.com with SMTP id b19so8753250pld.0 for ; Mon, 12 Oct 2020 08:10:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=cKT3zDC7RcD+qxlsarQtGU4Xfg6iOjI8Zr0WAOZjosk=; b=q3kErvyCPzm/XzTzLafXPlfSUiCGUG8v6Cr+By2Nv8/zKYkifgkbGzT1hjrgLgTxzO +infYIyp+rTTdA9RI9FGOS/Jcu7NX72q1eHEOQCxuizKxPpNGPg+ZuUNaGF5uimqLPl2 bYR5wXp+xXYecmiI4LYFxlO6deV1Zzj87AqZxy0NSU7pLHueMbTfm9XNutG34T7eIIzT U0dTI1Pp5wvNRAtGOhULdztt9GAmFo+ON7Dk6vznDLpC1En2dP5yFHwDa7UM+4g7b59H yqK0lGZSLUbeuFYzw4rSDGHSNkmfgBvXJCMQqeb55H7w4FDkEtkgNcH55CCjeg7utTti 8GrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=cKT3zDC7RcD+qxlsarQtGU4Xfg6iOjI8Zr0WAOZjosk=; b=iqzqhZSmTedMdY7+NIKJxjYSxwrsjtr2takj1ltAMCmyfUhYyAVQgXfCAwJGeKONei GRNE06rYqkUpHBiQ3MI1N4UXOxn3aKBC7Z5RLmVXcLbh0FNiOyTcw5c+9HwlWxBjdbIN BeTG+P4FRFae8jwWfmT7vOli1NrCg9HjBg1uRuLP3lyZ1IX1Nv5ew7NkwUr30yVv0jMh ACEu3z403W6Ln+H98aymkKuJhPvNI3DvYFnPDaQrUf+egyJzmb2sJ6OYbH64rZm/amQZ QbnEBfSF/cSzE8926LO2J+gMRfl5Q3ab8QUL7Tk+tkoC2vMWFWbLg0NFt9CaBuGPaxWJ +4tA== X-Gm-Message-State: AOAM531+qjjyCjVL4NNpXgIifddMJ850MzIBYSblunQS2e8k6kdlqg/a npyohW/b1Cc5JlhXqnMbPw/5M7SjEpCLniG/3lLMUG6bbG/L1Q== X-Google-Smtp-Source: ABdhPJxI+4ncgj35oue9MmfGyz6AP7oU3VC33xo3obZ9pCLdJ6Ip3NDG02uqqmNUdK5XGwfGoJi9TGf+w7NxfMSkMdc= X-Received: by 2002:a17:902:cd07:b029:d3:9be0:2679 with SMTP id g7-20020a170902cd07b02900d39be02679mr24419591ply.68.1602515395916; Mon, 12 Oct 2020 08:09:55 -0700 (PDT) MIME-Version: 1.0 References: <20201012094941.88333zq82rnh2lyc@www.staffmail.ed.ac.uk> In-Reply-To: <20201012094941.88333zq82rnh2lyc@www.staffmail.ed.ac.uk> From: Neal Richardson Date: Mon, 12 Oct 2020 08:09:44 -0700 Message-ID: Subject: Re: [R and C++] passing arrow::Arrow from R to C++ for reading and writing? To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000ebb6e405b17ab0c5" --000000000000ebb6e405b17ab0c5 Content-Type: text/plain; charset="UTF-8" Hi Colin, Does the code you shared run? If not, how does it fail? One guess is that you're probably getting undefined symbols errors because you need more than just -larrow. See https://github.com/apache/arrow/blob/master/r/configure#L35 for others you need, and depending on how you installed arrow, you likely also need -larrow_bundled_dependencies. Neal On Mon, Oct 12, 2020 at 1:49 AM Colin McLean wrote: > Dear Arrow users, > > I was wondering if anyone can help me understand how I can create an > arrow::Array object in R, then pass this into C++ (using the Rcpp > library) for both reading and writing too? Similar what is done using > the R bigmemory > (https://privefl.github.io/blog/Tip-Optimize-your-Rcpp-loops/) or > bigstatsr packages. > > Kindest Regards, > Colin Mclean. > > R script: > library(Rcpp) > library(arrow) > > ## compile c++ code > Sys.setenv("PKG_CXXFLAGS" = "-larrow") > sourceCpp("utils.cpp") > > N = 10 > X = arrow::Array$create(rep(0,N*N)) > > test( X$pointer() ) > > > utils.cpp: > // define headers > // [[Rcpp::plugins(cpp11)]] > // [[Rcpp::depends(arrow)]] > > #include > #include > #include > #include > #include > #include > #include > > > using namespace Rcpp; > using namespace std; > > // [[Rcpp::export]] > void test( XPtr aAPtr ){ > > cout << "read & write arrow::Array in test " << endl; > cout << aAPtr << endl; > > } > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > > --000000000000ebb6e405b17ab0c5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Colin,
Does the code you shared run? If = not, how does it fail?

One guess is that you'r= e probably getting undefined symbols errors because you need more than just= -larrow. See https://github.com/apache/arrow/blob/master/r/configure#L35 = for others you need, and depending on how you installed arrow, you likely a= lso need -larrow_bundled_dependencies.

Neal

On Mon, Oct 12, 2020 at 1:49 AM Colin McLean <cmclean5@staffmail.ed.ac.uk> wrote:
Dear Arrow users,

=C2=A0 I was wondering if anyone can help me understand how I can create an= =C2=A0
arrow::Array object in R, then pass this into C++ (using the Rcpp=C2=A0 library) for both reading and writing too? Similar what is done using=C2=A0=
the R bigmemory=C2=A0
(https://privefl.github.io/blog/Tip-Opti= mize-your-Rcpp-loops/) or=C2=A0
bigstatsr packages.

Kindest Regards,
Colin Mclean.

R script:
library(Rcpp)
library(arrow)

## compile c++ code
Sys.setenv("PKG_CXXFLAGS" =3D "-larrow")
sourceCpp("utils.cpp")

N =3D 10
X =3D arrow::Array$create(rep(0,N*N))

test( X$pointer() )


utils.cpp:
// define headers
// [[Rcpp::plugins(cpp11)]]
// [[Rcpp::depends(arrow)]]

#include <arrow/api.h>
#include <arrow/array.h>
#include <arrow/array/array_base.h>
#include <Rcpp.h>
#include <stdio.h>
#include <iostream>
#include <string>


using namespace Rcpp;
using namespace std;

// [[Rcpp::export]]
void test( XPtr<arrow::Array> aAPtr ){

=C2=A0 =C2=A0cout << "read & write arrow::Array in test &quo= t; << endl;
=C2=A0 =C2=A0cout << aAPtr << endl;

}

--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.


--000000000000ebb6e405b17ab0c5--