From user-return-833-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Wed Dec 9 11:27:33 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 900F018037A for ; Wed, 9 Dec 2020 12:27:33 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id C612545693 for ; Wed, 9 Dec 2020 11:27:32 +0000 (UTC) Received: (qmail 65895 invoked by uid 500); 9 Dec 2020 11:27:32 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 65885 invoked by uid 99); 9 Dec 2020 11:27:32 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Dec 2020 11:27:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 8C905C0426 for ; Wed, 9 Dec 2020 11:27:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: 0.199 X-Spam-Level: X-Spam-Status: No, score=0.199 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-fi.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=tickup-se.20150623.gappssmtp.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id pSjdxfwdLFKi for ; Wed, 9 Dec 2020 11:27:31 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::430; helo=mail-wr1-x430.google.com; envelope-from=anders.johansson@tickup.se; receiver= Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id D5DB67F9E0 for ; Wed, 9 Dec 2020 11:27:30 +0000 (UTC) Received: by mail-wr1-x430.google.com with SMTP id l9so1320341wrt.13 for ; Wed, 09 Dec 2020 03:27:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tickup-se.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=J+m9euUYoNf53A4DFCuolE1rAuPo6+lHv6WRl0EOQM4=; b=gKQM7H36r2nUQMKuiLO5xeDwu7TG4K33PGxBas/xvDrkv1sjOwBDz58v8Og5RoSY6Z 9VmNkYTSFSNpd1mu4+t3hUQGoI2UPvH0BuQLQn3JWmt02tRDFGj12UtsFNfYSJPw+kg0 GPik9n8mkl4KYDXtU/9lV9/E9vSv5mjROChmmydPMw6rvvGX4801kKj5JP30jHRSKv50 pzTaxawYynLF8Un5FWl7YHfmN7bEkIHRLXej4chwby+w9aqCirDtLgIuJLI7cBorLtse qG4X6uKGZiuQzBoXC0UXZ5xqrJIKzqds1l6Ry0Pa9G6UnZadrGKUAuMDcHAaAzX801qg Tk1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=J+m9euUYoNf53A4DFCuolE1rAuPo6+lHv6WRl0EOQM4=; b=N+yPkKeepDWTOP2leertAkjdzdq5atz1OhlhQi48hw12h/cOq7mLKDTk2g0Dhahe6h mhXTcwyvccfzs4ruHxUsAACboCvHSSKlTBaBtbN88jiArLaB9YWPOw05S9DueVs9dBW8 B2V3tXPHdGoBsawq7iPiK6ZXvTkWlKhB6UDAnVxzsl3fSnYK2NqXoWfgaa5ix0hUlAFU SE5PzeTtOU40TMNTYvyQJd3oaceGUsX9Q3x8D80QxrxE2oUPBypxP81ldEyo2ehrI/MH aPeYvq+mtIp5MfjwhFHn72O4hLNy3fMdeOsXrJP/8gWuKdNX2oTRlz8+gkM2TGoXXvcv 4gWg== X-Gm-Message-State: AOAM5319No+otCPQFeKlEJaex02u6dkIJGfSUq/KUNyRPWOwA60auB7b gUcNUtXaAOv3DqvS4ok5GqxDcS1Ksx/5TGEr5+9M188zbqw= X-Google-Smtp-Source: ABdhPJxRZPrPFrhHpzSHhS5rcUKnlvIL0khX/3+vaENBNnfDIoO/AZuQMbPVoDBnWOSbfuZ1q1JxUHNIdMaHFlVJk3k= X-Received: by 2002:adf:fa0c:: with SMTP id m12mr2235867wrr.222.1607513244365; Wed, 09 Dec 2020 03:27:24 -0800 (PST) MIME-Version: 1.0 References: <149eec92-e50d-4173-8c9d-89d7691f2bd2@www.fastmail.com> In-Reply-To: <149eec92-e50d-4173-8c9d-89d7691f2bd2@www.fastmail.com> From: anders johansson Date: Wed, 9 Dec 2020 12:27:13 +0100 Message-ID: Subject: Re: [C++] error when writing Timestamps in NANOS resolution using StreamWriter to parquet files To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000e708ae05b606571e" --000000000000e708ae05b606571e Content-Type: text/plain; charset="UTF-8" Hi, Thanks for pointing that out On Wed, Dec 9, 2020 at 11:20 AM Uwe L. Korn wrote: > Hello Anders, > > you have twice the same time_type in your mail. I guess one of them should > be different? > > Cheers > Uwe > > On Wed, Dec 9, 2020, at 11:00 AM, anders johansson wrote: > > Hi, > > I am trying to write time stamps in int64_t format representing time in > UTC normalized nanoseconds to a parquet file. > > I'm using the following code: > > auto time_type = LogicalType::Timestamp(true, > LogicalType::TimeUnit::NANOS, false, false); > NodeVector nv; > > nv.push_back(PrimitiveNode::Make("Time", Repetition::REQUIRED, time_type, > Type::INT64)); > > but when I try to write to the output stream > > std::shared_ptr parquet_os_; > *parquet_os_ << se.time; /* time is uint64_t */ > > I get the following runtime error:"Column converted type mismatch. Column > 'Time' has converted type[NONE] not 'INT_64'" > > Everything works fine if I set: > > auto time_type = LogicalType::Timestamp(true, > LogicalType::TimeUnit::NANOS, false, false); > > but I want it as Time or Timestamp so that I get it in the proper format > when I read the file using pandas in python. > > Thanks, > Anders > > > --000000000000e708ae05b606571e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

Thanks for pointing that= out

On Wed, Dec 9, 2020 at 11:20 AM Uwe L. Korn <uwelk@xhochy.com> wrote:
Hello Anders,
=

you have twice the same=C2=A0time_type in your mail. I = guess one of them should be different?

Cheers<= br>
Uwe

On Wed, Dec 9, 2020, at 11:00 AM= , anders johansson wrote:
Hi,

I am trying to write time stamps in int64_t format representing time in = UTC normalized nanoseconds to a parquet file.

= I'm using the following code:

auto time_ty= pe =3D LogicalType::Timestamp(true, LogicalType::TimeUnit::NANOS, false, fa= lse);
NodeVector nv;

nv.push_bac= k(PrimitiveNode::Make("Time", Repetition::REQUIRED,=C2=A0 time_ty= pe, Type::INT64));

but when I try to write to = the output stream

std::shared_ptr<parquet::= StreamWriter> parquet_os_;
*parquet_os_ << se.time; = /* time is uint64_t */

I get the following run= time error:"Column converted type mismatch.=C2=A0 Column 'Time'= ; has converted type[NONE] not 'INT_64'"

Everything works fine if I set:

auto t= ime_type =3D LogicalType::Timestamp(true, LogicalType::TimeUnit::NANOS, fal= se, false);

but I want it as Time or Timestamp= so that I get it in the proper format when I read the file using pandas in= python.

Thanks,
Anders

--000000000000e708ae05b606571e--