From user-return-598-archive-asf-public=cust-asf.ponee.io@arrow.apache.org  Mon Aug  3 01:42:23 2020
Return-Path: <user-return-598-archive-asf-public=cust-asf.ponee.io@arrow.apache.org>
X-Original-To: archive-asf-public@cust-asf.ponee.io
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153])
	by mx-eu-01.ponee.io (Postfix) with ESMTPS id 6A3F4180644
	for <archive-asf-public@cust-asf.ponee.io>; Mon,  3 Aug 2020 03:42:23 +0200 (CEST)
Received: from mail.apache.org (localhost [127.0.0.1])
	by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id 9BD3312465E
	for <archive-asf-public@cust-asf.ponee.io>; Mon,  3 Aug 2020 01:42:22 +0000 (UTC)
Received: (qmail 16135 invoked by uid 500); 3 Aug 2020 01:42:22 -0000
Mailing-List: contact user-help@arrow.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@arrow.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@arrow.apache.org>
List-Post: <mailto:user@arrow.apache.org>
List-Id: <user.arrow.apache.org>
Reply-To: user@arrow.apache.org
Delivered-To: mailing list user@arrow.apache.org
Received: (qmail 16125 invoked by uid 99); 3 Aug 2020 01:42:21 -0000
Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2020 01:42:21 +0000
Received: from localhost (localhost [127.0.0.1])
	by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 2D89B18147D
	for <user@arrow.apache.org>; Mon,  3 Aug 2020 01:42:21 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 0.049
X-Spam-Level:
X-Spam-Status: No, score=0.049 tagged_above=-999 required=6.31
	tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1,
	DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25,
	RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001,
	SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled
Authentication-Results: spamd3-us-west.apache.org (amavisd-new);
	dkim=pass (2048-bit key) header.d=gmail.com
Received: from mx1-ec2-va.apache.org ([10.40.0.8])
	by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024)
	with ESMTP id rpW3VhuXwxjp for <user@arrow.apache.org>;
	Mon,  3 Aug 2020 01:42:19 +0000 (UTC)
Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.217.50; helo=mail-vs1-f50.google.com; envelope-from=comicfans44@gmail.com; receiver=<UNKNOWN> 
Received: from mail-vs1-f50.google.com (mail-vs1-f50.google.com [209.85.217.50])
	by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id A13F2BE345
	for <user@arrow.apache.org>; Mon,  3 Aug 2020 01:42:19 +0000 (UTC)
Received: by mail-vs1-f50.google.com with SMTP id p8so10991349vsm.12
        for <user@arrow.apache.org>; Sun, 02 Aug 2020 18:42:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:from:date:message-id:subject:to;
        bh=YCscaD1eTtsinBQT92cbErTk0cd9iOlUtxG7nA4jvg8=;
        b=AXD5C/piLH1hR5397pdBcM8z2/JVbuFkco+1DMTf0LoqZljWU13HCLVaQ2FyjQUFE0
         Jfqzu8GYPEnNU9jzt21GkBIG71QgXcLVL1ZOsFp5v0SBfdiJPHJPts+C5tkj0c94MxPY
         F0oqOXU8fQOC/inrUvSoox8n7fVslpaHKQq4n0hXMHZQjpeu6fVfpDt0dcmzBZREaxNk
         h/zXAGVBLTo7VZT5f+8I7PQ0sLWch3cXy6TyZ2qRHcWS/pYzaofr31ax4eQELSGKJnk3
         2dKUXHKrtdgT2Y2uH2+3gE1HZa1DJOmtaEQXz7VOFGd7Io9eYRytAkzycZfVCTkI1OH1
         JoKQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
        bh=YCscaD1eTtsinBQT92cbErTk0cd9iOlUtxG7nA4jvg8=;
        b=J08IcnRFQ1YavvTc2O2MjsNf6v+XsH4I75ImdY8z+etUhZi8z3UNqqevE3/nTY78Kl
         yxPMw5zxEgMq6ckYIGKtBmOmkZXsFPKH+juj/LQ/sFZuf67GVHtNG9D+iujxSyLVU2gL
         MUVKoDqgT1RbhKZPY9Ez82icE8izFEYk1OBOrIW6j3o8QFoQ8lZy8zFfZ8PZcORT9tui
         iwvPVJpNI7a51H31IVUDHqk6YNt47ia6lVDPS9DKK91H341zkaR7/Y5oyb58sJyV3kcb
         TDCqHCygPwJM4JF4VFzJGZFHQWBr+st8SGy6GGj4Z+/ADOBBJsBrK59ZB5BwaCpZt5QF
         xcqw==
X-Gm-Message-State: AOAM530LVDmpTi97PZBcrBXUSHvBGj3CEdtVtuLVm6nv6O6bWvRO1hUT
	Ugp42/ghtRLpG3C0Aqgf3lFNmDNQ+l9Ub0TeTcXN3GROXMptjQ==
X-Google-Smtp-Source: ABdhPJzGXlgDyQRy7LXEHD58U+F0mfdvMwkieatgBwpylUPRneQ5GWWuD52m+l46UtSKq3b66HIqSV5JqjqhMmyAuow=
X-Received: by 2002:a67:ef1d:: with SMTP id j29mr10560777vsr.139.1596418932641;
 Sun, 02 Aug 2020 18:42:12 -0700 (PDT)
MIME-Version: 1.0
From: comic fans <comicfans44@gmail.com>
Date: Mon, 3 Aug 2020 09:42:01 +0800
Message-ID: <CAKzW8o=Hs09qJOHJqDoX8HT-AVn6ESrSvgqznZ=gDrybYSRTuA@mail.gmail.com>
Subject: question about read/write feather format and memory mapped write
To: user@arrow.apache.org
Content-Type: text/plain; charset="UTF-8"

Hello everyone,  I'm trying to write out a dataframe in feather format
from R and read it in C++,

my R code looks like this:

arrow::write_feather(data.frame(a=1:1000, b= 1000:1),
 'arrow.data', chunk_size=500, compression= 'uncompressed')

and my C++ code looks like this:

     auto column0 = table->column(0);
     for(int i=0; i< column0->num_chunks();++i){
         auto array = column0->chunk(i);
         auto buffers = array->data()->buffers;
         for(int j=0;j<buffers.size();++j){
             if(!buffers[j]){
                 std::cout<<j<<" null"<<std::endl;
             }else{
                 std::cout<<j<<" "<<buffers[j]->size()<<std::endl;
             }
         }
     }

I found that C++ read data buffer is
[nullptr,  500 number, nullptr, 500 number], if chunk_size =10,
I got [nullptr, 40 number, nullptr, 40 number ...] ,  which makes me
confusing , why a usefulless nullptr Buffer before every Buffer ?

another question is how to use arrow as a zero-copy TSDB, my
intention:

1. historic and new written data must be in contiguous memory ,
    can not be chunked (so I can't makes historic readonly part
     and newly writable part in different buffer)
2. historic data may be very big so I need it memory mapped
3. I also want to use memory map to persist new written data
   (don't have strict transaction requirements, OS scheduled flush
    is OK to me)
4. how many new data to write is known, so preallocated memory
    mapped file is OK.
5. all components live in same process, no cross-process
    communicate needed (so apache plasma not needed)
6. easily exchange data with R

firstly I think arrow is a good fit ,  but with some docs reading , I
realize the buffer in arrow can't be modified, if I a feather file
with array size preallocated, all data became readonly when reload
it (through memory mapped file interface) . I abuse arrow by
const cast the data pointer and write into it , since it's memory
mapped, modification do change the file as I intend, but I'd like
to know if there is better way to achieve my goal ? does arrow
 intend to support such usecase and I missed some API ?
any advise will be helpful.