From user-return-935-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Sun Jan 24 17:41:29 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 3423118064D for ; Sun, 24 Jan 2021 18:41:29 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 54F0C43A81 for ; Sun, 24 Jan 2021 17:41:28 +0000 (UTC) Received: (qmail 67494 invoked by uid 500); 24 Jan 2021 17:41:28 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 67484 invoked by uid 99); 24 Jan 2021 17:41:28 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 24 Jan 2021 17:41:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 514741FF39B for ; Sun, 24 Jan 2021 17:41:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: -0.001 X-Spam-Level: X-Spam-Status: No, score=-0.001 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id vH-cCA0rmTE8 for ; Sun, 24 Jan 2021 17:41:26 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::22e; helo=mail-lj1-x22e.google.com; envelope-from=wesmckinn@gmail.com; receiver= Received: from mail-lj1-x22e.google.com (mail-lj1-x22e.google.com [IPv6:2a00:1450:4864:20::22e]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id CE17F7FA16 for ; Sun, 24 Jan 2021 17:41:26 +0000 (UTC) Received: by mail-lj1-x22e.google.com with SMTP id f17so12473394ljg.12 for ; Sun, 24 Jan 2021 09:41:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=nfyEBoZSroengOYCORskRK/8TYNUIhkreAOatME1Fe4=; b=KX2aP1xwLtDF9W5LAshmxl4dWRUXe4ZNdOikk694SUydKj5cbtraw3x/+tzm4wNhid 17fQF4Ury+kD3csCz0EJyn8PyaACoB0RjqYuadn4XtzTxnvsO54TcjIWsPOCIvSql85N RBjch4UYFRudrpkMJ1qA+tGNNoAHaNdjGRY4kLVf+p8Ia2W+Iiiuv1bygyGeBVKGXhrv /2I58PdBmOaHY0sINQvQjR9tniCt+fS+rYgizv8vqK2T0xqPvDVD5/9idSpFb55Lo7/T 91+Sp0oPHXOqetjXOrQENfV8yxdiavIkspo4UitZPew/Wn+U6FHMoiOOm2XHRXAAYBzg 7ZTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=nfyEBoZSroengOYCORskRK/8TYNUIhkreAOatME1Fe4=; b=cqqsZMp2/k1rGYYy1iYXIAXEd9yvWYlkHBrDcytwwHj9lxe3b47U7xD9cGWLVWCiz3 sdSc7t7Te/E9A0ksSZm+AvWMsLFim4rWuUSLWfwezlFmSyttkhw19rXjpcURVq5BVVTH 4LM3L4esxgneqLHP0jRn5ECVcqGMXgAQdk4gc18cHNg0+bekz1PaxkF6VSR9aVyYZVsC 7HlR4CS6vLuaGibDVYDic3F979kX8AYTH91MeDtzmebgc0HFjjWo1cdSg21l0ofB7mS5 Xv5/rHJimRtGxqL0334xOEonnjxL2lBPAAKMYMfmkwI24x1C7ee/tqenHnsg0X/9uZQx q83g== X-Gm-Message-State: AOAM533vKLdQIe41FP2q94HqeP5easTroZVcimYjRo8RkOgKiNN4ZrKL wBUc5vfaW5J/G1NuvIGAJ8+ptxBqtN6//lQ+fXhPRdA2yPM= X-Google-Smtp-Source: ABdhPJxmL/xAC1kQc/bARraOl5XAshFYoqQrL2iUFsyEa8IIMvyNuLEt4J9xBildiJ8amFjGhLbL0qcfLQsbyqaMvac= X-Received: by 2002:a05:651c:1107:: with SMTP id d7mr267231ljo.10.1611510079523; Sun, 24 Jan 2021 09:41:19 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Wes McKinney Date: Sun, 24 Jan 2021 11:41:08 -0600 Message-ID: Subject: Re: [python] not an arrow file To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000d78cd005b9a8edb9" --000000000000d78cd005b9a8edb9 Content-Type: text/plain; charset="UTF-8" Can you show your C++ code? On Sun, Jan 24, 2021 at 8:10 AM Teh, Kenneth M. wrote: > Just started with arrow... > > I wrote a record batch to a file using ipc::MakeFileWriter to create a > writer and writer->WriteRecordBatch in a C++ program and tried to read it > in python with: > > [] import pyarrow as pa > [] reader = pa.ipc.open_file("myfile") > > > It raises the ArrowInvalid with the message "not an arrow file". > > If I write it out as a Table in feather format, I can read it in python. > But I want to write large files on the order of 100GB or more and then read > them back into python as pandas dataframes or something similar. > > So, I switched to using an ipc writer. > > Can something point me in the right direction? Thanks. > > Ken > --000000000000d78cd005b9a8edb9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Can you show your C++ code?

On Sun, Jan 24, 2021 at 8= :10 AM Teh, Kenneth M. <teh@anl.gov&g= t; wrote:
Just started with arrow...

I wrote a record batch to a file using ipc::MakeFileWriter to create a writ= er and writer->WriteRecordBatch in a C++ program and tried to read it in= python with:

[] import pyarrow as pa
[] reader =3D pa.ipc.open_file("myfile")


It raises the ArrowInvalid with the message "not an arrow file".<= br>

If I write it out as a Table in feather format, I can read it in python. Bu= t I want to write large files on the order of 100GB or more and then read t= hem back into python as pandas dataframes or something similar.

So, I switched to using an ipc writer.

Can something point me in the right direction?=C2=A0 Thanks.

Ken
--000000000000d78cd005b9a8edb9--