From user-return-835-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Wed Dec 9 11:36:59 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-ec2-va.apache.org (mxout1-ec2-va.apache.org [3.227.148.255]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 3E6CD18037A for ; Wed, 9 Dec 2020 12:36:59 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-ec2-va.apache.org (ASF Mail Server at mxout1-ec2-va.apache.org) with SMTP id 7B63E456A9 for ; Wed, 9 Dec 2020 11:36:58 +0000 (UTC) Received: (qmail 79739 invoked by uid 500); 9 Dec 2020 11:36:58 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 79729 invoked by uid 99); 9 Dec 2020 11:36:58 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Dec 2020 11:36:58 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 77CCE1FF42D for ; Wed, 9 Dec 2020 11:36:57 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: 0.199 X-Spam-Level: X-Spam-Status: No, score=0.199 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.2, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=tickup-se.20150623.gappssmtp.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id 8inMYAsS7Wrh for ; Wed, 9 Dec 2020 11:36:56 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::335; helo=mail-wm1-x335.google.com; envelope-from=anders.johansson@tickup.se; receiver= Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id B2E9C7F9E4 for ; Wed, 9 Dec 2020 11:36:56 +0000 (UTC) Received: by mail-wm1-x335.google.com with SMTP id a3so1303491wmb.5 for ; Wed, 09 Dec 2020 03:36:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tickup-se.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=oYzA2wZ7+TLyICP8GNjJMSQEWWJI93vodQU145p665U=; b=NWDncNSYiyacrvV89RxqGTfiGNlIMaA8j0CTJVHF8ZWaAdRpVbpcLlna+l2+6U8M1e PSHnEWar7/VQvTR+CfEKxLDK4XpuDEMoHHt33Ek879maf43mMT+CcF/0ngyadnRoDXUQ BXLRLY1pixRdlF9PkaTC5R38E6ol+5JKkq+24H7h+6YGooAhTNONvkfoOeVRqNk2rlGA E4UK4mgM7NzsKD6Bobot3647MlS3Jy2hIN/hM6NC33tWXlRYkyCT2yBDAUMJxBHS5/Lj cFk9IFXIjCZsrb1ppvN7bbsSTqnhoVGOlHr60FsR1g39xsFWg02NStLnWxLRbMwlhGqH mW+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=oYzA2wZ7+TLyICP8GNjJMSQEWWJI93vodQU145p665U=; b=UP9f8sBQrzqltWaPZTUIZe7O3dx6MgixsxRKKIxTfVskqSD1Ui8QGz0SsvqU3zWbU7 PxPLK0T0duA/oJj7U7zNFC3fURZOWz/1O1YwFs1FxtzSde6uJqupitQQVycYSd0yqWqx CcKbhCs7JDPt0EuvFw30MB2ANmHK3ZxhWc9vVmC5WFLAeCXlf7ACXQlnsGA1brpK9Y+q XriFdFB6HPSfiJM99uYQDe9BjOi+hrFl5cryRo5ld58Xf/YZzN5MmtC+ke76okmYkD+m GKLN7CvoqX1wgaADcmHSGmSd96bmw8U1wir9efnZdsl4t14Eni1cEU6sI4PPJez6NyIo FUAQ== X-Gm-Message-State: AOAM533CAdkpbnehSVvrogJqfb+VqHZeK8AxnEKQqX8l99rZfeacttX8 rdSj06J1qnneGp6nGgjJWHQzjMYbgRK82nt6KdKd3Z73fh1fEg== X-Google-Smtp-Source: ABdhPJytlS+Ighfh3iv2xodsc13lZhZwgabTX5OdmIP7xL1EN1MSidXZ73R6TnbIUWr8dQe+h1p5I9mYy20o+sTiil4= X-Received: by 2002:a1c:186:: with SMTP id 128mr2324588wmb.176.1607513816094; Wed, 09 Dec 2020 03:36:56 -0800 (PST) MIME-Version: 1.0 References: <149eec92-e50d-4173-8c9d-89d7691f2bd2@www.fastmail.com> In-Reply-To: From: anders johansson Date: Wed, 9 Dec 2020 12:36:45 +0100 Message-ID: Subject: Re: [C++] error when writing Timestamps in NANOS resolution using StreamWriter to parquet files To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="000000000000faeeae05b6067934" --000000000000faeeae05b6067934 Content-Type: text/plain; charset="UTF-8" Hi again, I ran into a similar problem with decimal, where if I set the type to LogicalType::Decimal(4, 4); and try to write a uint32_t I get the following error message: "Column converted type mismatch. Column 'Price' has converted type[DECIMAL] not 'INT_32'" When I look at the StreamWriter code (line 186 in stream_writer.cc), it looks like the function CheckColumn will throw the error whenever a non trivial type is written to the output stream. As I understand it from the code comments, the converted type is legacy code, so I guess this check is outdated? BR, Anders On Wed, Dec 9, 2020 at 12:27 PM anders johansson wrote: > It should be > auto time_type = LogicalType::Int(64, true); > > On Wed, Dec 9, 2020 at 12:27 PM anders johansson < > anders.johansson@tickup.se> wrote: > >> Hi, >> >> Thanks for pointing that out >> >> On Wed, Dec 9, 2020 at 11:20 AM Uwe L. Korn wrote: >> >>> Hello Anders, >>> >>> you have twice the same time_type in your mail. I guess one of them >>> should be different? >>> >>> Cheers >>> Uwe >>> >>> On Wed, Dec 9, 2020, at 11:00 AM, anders johansson wrote: >>> >>> Hi, >>> >>> I am trying to write time stamps in int64_t format representing time in >>> UTC normalized nanoseconds to a parquet file. >>> >>> I'm using the following code: >>> >>> auto time_type = LogicalType::Timestamp(true, >>> LogicalType::TimeUnit::NANOS, false, false); >>> NodeVector nv; >>> >>> nv.push_back(PrimitiveNode::Make("Time", Repetition::REQUIRED, >>> time_type, Type::INT64)); >>> >>> but when I try to write to the output stream >>> >>> std::shared_ptr parquet_os_; >>> *parquet_os_ << se.time; /* time is uint64_t */ >>> >>> I get the following runtime error:"Column converted type mismatch. >>> Column 'Time' has converted type[NONE] not 'INT_64'" >>> >>> Everything works fine if I set: >>> >>> auto time_type = LogicalType::Timestamp(true, >>> LogicalType::TimeUnit::NANOS, false, false); >>> >>> but I want it as Time or Timestamp so that I get it in the proper format >>> when I read the file using pandas in python. >>> >>> Thanks, >>> Anders >>> >>> >>> --000000000000faeeae05b6067934 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi again,

I ran into a simil= ar problem with decimal, where if I set the type to LogicalType::Decimal(4,= 4); and try to write a uint32_t I get the following error message: "C= olumn converted type mismatch.=C2=A0 Column 'Price' has converted t= ype[DECIMAL] not 'INT_32'"

When = I look at the StreamWriter code (line 186 in stream_writer.cc), it looks li= ke the function CheckColumn will throw the error whenever a non trivial typ= e is written to the output stream.

As I under= stand it from the code comments, the converted type is legacy code, so I gu= ess this check is outdated?

BR,
Ande= rs

On Wed, Dec 9, 2020 at 12:27 PM anders johansson <anders.johansson@tickup.se> wrot= e:
It should be
=C2=A0 auto time_type =3D LogicalType:= :Int(64, true);

On Wed, Dec 9, 2020 at 12:27 PM anders johansson &= lt;anders.j= ohansson@tickup.se> wrote:
Hi,

Thanks= for pointing that out

On Wed, Dec 9, 2020 at 11:20 AM Uwe L. Korn &= lt;uwelk@xhochy.com> wrote:
Hello Anders,

you have twice th= e same=C2=A0time_type in your mail. I guess one of them should be different= ?

Cheers
Uwe

On Wed, Dec 9, 2020, at 11:00 AM, anders johansson wrote:
Hi,

I am trying to write time stamps in int64_t format re= presenting time in UTC normalized nanoseconds to a parquet file.
<= div>
I'm using the following code:

auto time_type =3D LogicalType::Timestamp(true, LogicalType::TimeUni= t::NANOS, false, false);
NodeVector nv;

nv.push_back(PrimitiveNode::Make("Time", Repetition::REQU= IRED,=C2=A0 time_type, Type::INT64));

but when= I try to write to the output stream

std::shar= ed_ptr<parquet::StreamWriter> parquet_os_;
*parquet_os_= << se.time; /* time is uint64_t */

I ge= t the following runtime error:"Column converted type mismatch.=C2=A0 C= olumn 'Time' has converted type[NONE] not 'INT_64'"

Everything works fine if I set:
auto time_type =3D LogicalType::Timestamp(true, LogicalType::T= imeUnit::NANOS, false, false);

but I want it a= s Time or Timestamp so that I get it in the proper format when I read the f= ile using pandas in python.

Thanks,
Anders

=
--000000000000faeeae05b6067934--