From user-return-15-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Wed Oct 3 18:00:47 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 92D7218065B for ; Wed, 3 Oct 2018 18:00:46 +0200 (CEST) Received: (qmail 68679 invoked by uid 500); 3 Oct 2018 16:00:45 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 68669 invoked by uid 99); 3 Oct 2018 16:00:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Oct 2018 16:00:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 189A81A018D for ; Wed, 3 Oct 2018 16:00:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.114 X-Spam-Level: *** X-Spam-Status: No, score=3.114 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, MIME_BOUND_DIGITS_15=1.225, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id TkP50lAYzM3H for ; Wed, 3 Oct 2018 16:00:44 +0000 (UTC) Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id AC4465F433 for ; Wed, 3 Oct 2018 16:00:43 +0000 (UTC) Received: by mail-pf1-f180.google.com with SMTP id s5-v6so1895554pfj.7 for ; Wed, 03 Oct 2018 09:00:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=T98tyK4Jr1lIVRjIzEsklMPwfeszuv8pVfiTtirbx5E=; b=HyBchvWUSCLydKfS9yDZa3Zj0UNtHiMQa130G0TpCWK/XhWHbbwNaJSCR4d1uiv+7E DdyBh5f81In7jdOhl21EQMiW+X48IRYxEDNMb4YfsqqgcDsB+AfoyXRqlqg3JLWkeGLE bFX+q1SpuanSNsd68v9q7SwXGtb5xvwBgBao5KmJw+iie5w6csoDZjAOknXk7uHJkOfa zA5OvgKC0UmEssFiC3uKI9gS0ALAGPeu2cSZpuOF1YTy3D+uzwmGFpFNp/S8z2pNhffA 2EPVeTkQ2U55mdGRKG96VvUEYV+wvr6Cb7DG96GTptLDjDPQttsfFLpLDakPqp1AF0iu HdQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=T98tyK4Jr1lIVRjIzEsklMPwfeszuv8pVfiTtirbx5E=; b=npEB33MUVLURt+UvbizpbdGShpyw4RMJV++jGCI3k6ozVLDG3Tl7n6a1RzI5NArIi+ fskVNBvl/nQOoP7JU5dHbZJlCKX09UCVjyXw9bjy1hy9v4KrsBEEZwA4+4aRfDf/LObm MyOF//+URrtkJW/JykIM1yuJsF1vz2jnEe48DTQC0gfNvfdb7Gs0m9vKHVnRg52+tr20 hDIufuA6AHTXahRxiBKewfU+y98Rk6UUeJ613a8oVmpHom7mEA05CfIm0MIvCh4FU4Ln HLxniPKFJL54psFnKru4iEsZITmi53SjvkG8K6Tx5ZBLeR/JbzeNMXkp3prFCsUmX1vP fCig== X-Gm-Message-State: ABuFfojKPlYD+M6KpPsHD85RrTA4dxcyPcKw8/pDkbabl3q0ZiE2PFM0 53pi0nMN4DlyywCVXeQ3Y/Z9q9MKvfpI2BtqRuCzab03 X-Google-Smtp-Source: ACcGV606kTL29I0chdC+fv4VRxhSrnGEisIZ3zkM97iUpjxlV/LqpipIboBUlZzmx5YO/sKI3j7XywXsTyEytWUrdn4= X-Received: by 2002:a62:670a:: with SMTP id b10-v6mr2246653pfc.243.1538582436600; Wed, 03 Oct 2018 09:00:36 -0700 (PDT) MIME-Version: 1.0 From: Bipin Mathew Date: Wed, 3 Oct 2018 12:00:25 -0400 Message-ID: Subject: Help with writing Apache Arrow tables to shared memory. To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="0000000000009762030577552374" --0000000000009762030577552374 Content-Type: text/plain; charset="UTF-8" Good Morning Everyone, I originally posted this question to the dev channel, not knowing a user channel was available. This channel is more probably more appropriate and I am hoping the kind souls here can help me. How, fundamentally, are we expected, to copy or indeed directly write a arrow table to shared memory using the cpp sdk? Currently, I have an implementation like this: 77 std::shared_ptr B; > 78 std::shared_ptr buffer; > 79 std::shared_ptr writer; > 80 arrow::MemoryPool* pool = arrow::default_memory_pool(); > 81 arrow::io::BufferOutputStream::Create(4096,pool,&buffer); > 82 std::shared_ptr table; > 83 karrow::ArrowHandle *h; > 84 h = (karrow::ArrowHandle *)Kj(khandle); > 85 table = h->table; > 86 > 87 > arrow::ipc::RecordBatchStreamWriter::Open(buffer.get(),table->schema(),&writer); > 88 writer->WriteTable(*table); > 89 writer->Close(); > 90 buffer->Finish(&B); > 91 > 92 // printf("Investigate Memory usage."); > 93 // getchar(); > 94 > 95 > 96 std::shared_ptr mm; > 97 > arrow::io::MemoryMappedFile::Create("/dev/shm/arrow_table",B->size(),&mm); > 98 mm->Write(B->data(),B->size()); > 99 mm->Close(); "table" on line 85 is a shared_ptr to a arrow::Table object. As you can see there, I write to an arrow:Buffer then write that to a memory mapped file. Is there a more direct approach? I watched this video of a talk @Wes McKinney gave here: https://www.dremio.com/webinars/arrow-c++-roadmap-and-pandas2/ Where a method: arrow::MemoryMappedBuffer was referenced, but I have not seen any documentation regarding this function. Has it been deprecated? Also, as I mentioned, "table" up there is a arrow::Table object. I create it columnwise using various arrow::[type]Builder functions. Is there anyway to actually even write the original table directly into shared memory? Any guidance on the proper way to do these things would be greatly appreciated. Regards, Bipin --0000000000009762030577552374 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Good Morning Everyone,

=C2=A0 =C2=A0 I = originally posted this question to the dev channel, not knowing a user chan= nel was available. This channel is more probably more appropriate and I am = hoping the kind souls here can help me. How, fundamentally, are we expected= , to copy or indeed directly write a arrow table to shared memory using the= cpp sdk? Currently, I have an implementation like this:

=C2=A077=C2=A0 =C2= =A0std::shared_ptr<arrow::Buffer> B;
=C2=A078=C2=A0 =C2=A0std::sha= red_ptr<arrow::io::BufferOutputStream> buffer;
=C2=A079=C2=A0 =C2= =A0std::shared_ptr<arrow::ipc::RecordBatchWriter> writer;
=C2=A080= =C2=A0 =C2=A0arrow::MemoryPool* pool =3D arrow::default_memory_pool();
= =C2=A081=C2=A0 =C2=A0arrow::io::BufferOutputStream::Create(4096,pool,&b= uffer);
=C2=A082=C2=A0 =C2=A0std::shared_ptr<arrow::Table> table;<= br>=C2=A083=C2=A0 =C2=A0karrow::ArrowHandle *h;
=C2=A084=C2=A0 =C2=A0h = =3D (karrow::ArrowHandle *)Kj(khandle);
=C2=A085=C2=A0 =C2=A0table =3D h= ->table;
=C2=A086=C2=A0
=C2=A087=C2=A0 =C2=A0arrow::ipc::RecordBat= chStreamWriter::Open(buffer.get(),table->schema(),&writer);
=C2= =A088=C2=A0 =C2=A0writer->WriteTable(*table);
=C2=A089=C2=A0 =C2=A0wr= iter->Close();
=C2=A090=C2=A0 =C2=A0buffer->Finish(&B);
=C2= =A091=C2=A0
=C2=A092=C2=A0 =C2=A0// printf("Investigate Memory usag= e.");
=C2=A093=C2=A0 =C2=A0// getchar();
=C2=A094=C2=A0
=C2= =A095=C2=A0
=C2=A096=C2=A0 =C2=A0std::shared_ptr<arrow::io::MemoryMap= pedFile> mm;
=C2=A097=C2=A0 =C2=A0arrow::io::MemoryMappedFile::Create= ("/dev/shm/arrow_table",B->size(),&mm);
=C2=A098=C2=A0 = =C2=A0mm->Write(B->data(),B->size());
=C2=A099=C2=A0 =C2=A0mm-&= gt;Close();

"table" on line= 85 is a shared_ptr to a arrow::Table object. As you can see there, I write= to an arrow:Buffer then write that to a memory mapped file. Is there a mor= e direct approach? I watched this video of a talk=C2=A0@Wes McKinney gave h= ere:


Where = a method:=C2=A0arrow::MemoryMappedBuffer was referenced, but I have not see= n any documentation regarding this function. Has it been deprecated?=C2=A0<= /div>

Also, as I mentioned, "table" up there i= s a arrow::Table object. I create it columnwise using various arrow::[type]= Builder functions. Is there anyway to actually even write the original tabl= e directly into shared memory? Any guidance on the proper way to do these t= hings would be greatly appreciated.

Regards,
=

Bipin
--0000000000009762030577552374--