From user-return-130-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Mon May 20 14:22:13 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8417A18076D for ; Mon, 20 May 2019 16:22:13 +0200 (CEST) Received: (qmail 17542 invoked by uid 500); 20 May 2019 14:22:12 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 17532 invoked by uid 99); 20 May 2019 14:22:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 May 2019 14:22:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CA1D7180E6A for ; Mon, 20 May 2019 14:22:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.003 X-Spam-Level: ** X-Spam-Status: No, score=2.003 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=353solutions-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 1iEWarnknL5I for ; Mon, 20 May 2019 14:22:05 +0000 (UTC) Received: from mail-oi1-f194.google.com (mail-oi1-f194.google.com [209.85.167.194]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id ADA1D6119B for ; Mon, 20 May 2019 14:22:04 +0000 (UTC) Received: by mail-oi1-f194.google.com with SMTP id u64so879836oib.1 for ; Mon, 20 May 2019 07:22:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=353solutions-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=fLxOV0zSgDiLKrgj/itHeEAqdRrd/b+fLWUtgl86jnM=; b=0Hc699eMginpCp43UuKh7O06h4HlYfhIbR48rhDrD3CczWhg/pvKLim2OJuByDhzU0 ktXwNK3WXIA+nBROyMiJpamM3dVfR2tjAhhbl+1HcxsftgKayiHWb17lqPaUIoVTGCJp qmeMq23UiU3S+fkE+kK1ZVGjAm1i7JH8gUkeLgJB/K7gz5wn6YaOXGqPib0Avu5nx2pJ vVAF2oORSXitBdndR8myVMuPZE6ExTOCSzhlPTIsbGNyGORz2gQo62pVDmrOKzUxNg1S u65l6mWxbewfv0SDWaXFQ+E6UpJoCx0hwljV8lI+B9sM6r0xt6ml3XorQZQtpatB6YeQ Y1uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=fLxOV0zSgDiLKrgj/itHeEAqdRrd/b+fLWUtgl86jnM=; b=N/Qkxz1ku57KhpM8BDeHlehzHCpuCCMwPKmlO+U8p6lai43IZ5CWQRobPahZD8tEr2 TNt4OopF5Q4pgEbpB+yZlZDMQTdnkmRsP3Qa3YOIt8iqogkA6oppgR37jZ8QOVaoeYZa RJJpf764FoNtoivynnjK7ujXpFQ6ImfAgYEyPkqsfmmBiqehWXzrjyHrko4rMh/DIzg7 bud1sxOJSqG1CDivXUoOCTEi190SeKmPDNj4BupR4iseMxGpyQgWmxnMZ4v0LfeCCQnx R674r4gLX8t/hllUZs1teQQLNusV+Pe/z3/zGWK4by9BnqrFh4u+LAOxx4S69E5ltOru GSzA== X-Gm-Message-State: APjAAAUrIXisuNTLTaSxMK3R19I67cbULojhFC39IxqwMeHpHfA9a2U/ u5Yn8MmXKsa3AXjziXUi/qcYdfpOifWOngLqohz3nnXN X-Google-Smtp-Source: APXvYqzoGgnEVPLZ+FOPGT1h5fOgDlHqOhPzgNgZD/q31vg7/Qz5tmSbZV6YQLvS0id0pzxM5wiHW9t33TLuhvkTPCc= X-Received: by 2002:aca:bac3:: with SMTP id k186mr13799804oif.160.1558362123168; Mon, 20 May 2019 07:22:03 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Miki Tebeka Date: Mon, 20 May 2019 17:21:52 +0300 Message-ID: Subject: Re: [C++] Storing/retreiving a Table in plasma To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000c88539058952741f" --000000000000c88539058952741f Content-Type: text/plain; charset="UTF-8" Thanks Wes On Mon, May 20, 2019 at 4:24 PM Wes McKinney wrote: > See https://issues.apache.org/jira/browse/ARROW-5377 > > On Mon, May 20, 2019 at 8:15 AM Wes McKinney wrote: > > > > hi Miki, > > > > Steps > > > > * Convert the Table to a sequence of RecordBatch objects. You can use > > arrow::TableBatchReader to do this [1] > > * Write a stream using MockOutputStream [2] > > * Use the reported size of the total stream to allocate memory in Plasma > > * Write a real stream using arrow::io::FixedSizeBufferWriter > > > > I'm interested at some point to reduce the amount of boilerplate > > associated with this process, and also to avoid multiple metadata > > serialization and record batch disassembly steps. I'll open a JIRA > > issue > > > > We'd be delighted if you would contribute to the C++ documentation at > > https://github.com/apache/arrow/tree/master/docs/source/cpp > > > > - Wes > > > > [1]: > https://github.com/apache/arrow/blob/master/cpp/src/arrow/table.h#L340 > > [2]: > https://github.com/apache/arrow/blob/7a5562174cffb21b16f990f64d114c1a94a30556/cpp/src/arrow/io/memory.h#L89 > > > > On Mon, May 20, 2019 at 7:24 AM Miki Tebeka > wrote: > > > > > > Hi, > > > > > > I'm looking for an example on how to store/retrieve a an arrow::Table > in plasma. The examples I see in the documentation site are for basic types. > > > > > > My end goal is to create data (Table) in C++, store it in plasma and > read if from Python. > > > > > > From reading around, I need to allocate buffer in plasma, but how can > I find the size of the Table to allocate the table? And how can I serialize > it into the created Buffer? > > > > > > Thanks, > > > Miki > --000000000000c88539058952741f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks Wes

On Mon, May 20, 2019 at 4:24 PM Wes = McKinney <wesmckinn@gmail.com= > wrote:
See = https://issues.apache.org/jira/browse/ARROW-5377=

On Mon, May 20, 2019 at 8:15 AM Wes McKinney <wesmckinn@gmail.com> wrote:
>
> hi Miki,
>
> Steps
>
> * Convert the Table to a sequence of RecordBatch objects. You can use<= br> > arrow::TableBatchReader to do this [1]
> * Write a stream using MockOutputStream [2]
> * Use the reported size of the total stream to allocate memory in Plas= ma
> * Write a real stream using arrow::io::FixedSizeBufferWriter
>
> I'm interested at some point to reduce the amount of boilerplate > associated with this process, and also to avoid multiple metadata
> serialization and record batch disassembly steps. I'll open a JIRA=
> issue
>
> We'd be delighted if you would contribute to the C++ documentation= at
> https://github.com/apache/arrow/tree= /master/docs/source/cpp
>
> - Wes
>
> [1]: https://github.com/a= pache/arrow/blob/master/cpp/src/arrow/table.h#L340
> [2]: https://github.com/apache/arrow/blob/7a5562174cffb21b16f99= 0f64d114c1a94a30556/cpp/src/arrow/io/memory.h#L89
>
> On Mon, May 20, 2019 at 7:24 AM Miki Tebeka <miki@353solutions.com> wrote: > >
> > Hi,
> >
> > I'm looking for an example on how to store/retrieve a an arro= w::Table in plasma. The examples I see in the documentation site are for ba= sic types.
> >
> > My end goal is to create data (Table) in C++, store it in plasma = and read if from Python.
> >
> > From reading around, I need to allocate buffer in plasma, but how= can I find the size of the Table to allocate the table? And how can I seri= alize it into the created Buffer?
> >
> > Thanks,
> > Miki
--000000000000c88539058952741f--