From user-return-855-archive-asf-public=cust-asf.ponee.io@arrow.apache.org Thu Dec 17 18:10:07 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 0F70E180608 for ; Thu, 17 Dec 2020 19:10:07 +0100 (CET) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 176C16723E for ; Thu, 17 Dec 2020 18:10:05 +0000 (UTC) Received: (qmail 34040 invoked by uid 500); 17 Dec 2020 18:10:05 -0000 Mailing-List: contact user-help@arrow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@arrow.apache.org Delivered-To: mailing list user@arrow.apache.org Received: (qmail 34030 invoked by uid 99); 17 Dec 2020 18:10:05 -0000 Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Dec 2020 18:10:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id 659AD1FF39A for ; Thu, 17 Dec 2020 18:10:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: 0.702 X-Spam-Level: X-Spam-Status: No, score=0.702 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.2, KAM_NUMSUBJECT=0.5, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=achren-org.20150623.gappssmtp.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id 24N6JwMSvfWE for ; Thu, 17 Dec 2020 18:10:03 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::32b; helo=mail-wm1-x32b.google.com; envelope-from=nite@achren.org; receiver= Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id AC19A7FA00 for ; Thu, 17 Dec 2020 18:10:03 +0000 (UTC) Received: by mail-wm1-x32b.google.com with SMTP id 3so6503924wmg.4 for ; Thu, 17 Dec 2020 10:10:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=achren-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=4KujNqpBZjTRtiWFDAKX60GMGUlXunRC1NRsJO7bzqM=; b=dnhdEF2NAOVIMkk7e24UyvPJQ/lgLN1LZRQoZmqLDs0uMA28Yw8S95/jRmxBkATd1y x18k0Di3dx40YReWvj+FLK/VNGgFhHp1pqACc5XhhHHpyn9jPUxbgQPQ7Nu2ZLgi4gr5 9mU4Hd+YwkOi63lRfQ9IdMzeQZZgLwjdnzvIqn1CpWmC4t0n6UDuVql/eRw0KYXuCs+y RrPU7cRoaVlKIPEv+F+6YQtlClSMTfa0B8EJynidmTHByV4jGFKNLuALjGd43aTiC33B 8pRtDyWxr/zT7k+RG1YhrBT1Seyn3u9oCnjllzUGOFrm30vWgkogPubr+IcsdeiPhrbX NYtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=4KujNqpBZjTRtiWFDAKX60GMGUlXunRC1NRsJO7bzqM=; b=aeEZ+7laBk4P6+OL89poEjutthiVKF9QchJGFFr+iADEmy59hnpEXDxCZ88IQnZf1L pLfqBnQ+ez1zCtCqOucdo8zamT9cpLX63K6BDaPT6NyZndrlsem6WpGC3tQD7Zqqu/Pq w/gguSAgPzwxctR066lIhRSpCSShcPWRoD5R++mfIZnV2xqzYn7jJ7RuVlKqSF2GrMId oR1R0nC7W06JZ4dT2Zw/F+Fu3/OyPSFMzR5COqcbFEd77pUfUdHBdjgVxNk7cKnj55mA vw+BmLqjL9HGBMQHTWKzYo/OIuYmLWamGIMo1uJ+ieWtC3gEW7Rxdlk16R5CduS6kQCA nzPQ== X-Gm-Message-State: AOAM532qKTYQMzqHetpI86y3pqN+a/GjnrnCRDSCr6Fo5382CA+1vdLt Fc06NRx6VBQ1kruQZ2h8s0+qFV4A+gFqgQw9bRvUIHB94CQvy0KzP5w= X-Google-Smtp-Source: ABdhPJyiatULGu+FtkKO/M3+fFkjSVtW8S6Bsl/Cn5r+83EZ46Zia74jPlAESPsCWLenYmuzYkY2jJ5tVrvO4aR80Fg= X-Received: by 2002:a1c:6856:: with SMTP id d83mr533007wmc.119.1608228596716; Thu, 17 Dec 2020 10:09:56 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Andrew Clancy Date: Thu, 17 Dec 2020 18:09:20 +0000 Message-ID: Subject: Re: [javascript] cant get timestamps in arrow 2.0 To: user@arrow.apache.org Content-Type: multipart/alternative; boundary="00000000000039aebd05b6ace648" --00000000000039aebd05b6ace648 Content-Type: text/plain; charset="UTF-8" So, I figured out the issue here - I had to remove compression from the pyarrow feather.write_feather(compression='uncompressed'). Is there any way to read a compressed feather file in arrow js? See the comment under the first answer here: https://stackoverflow.com/questions/64629670/how-to-write-a-pandas-dataframe-to-arrow-file/64648955#64648955 I couldn't find anything in the arrow docs or notebooks on this - I'm assuming that's related to javascript compression libraries being so limited. On Mon, 14 Dec 2020 at 21:32, Andrew Clancy wrote: > Hi all, > > I have a simple feather file created via a pandas to_feather with a > datetime64[ns] column, and cannot get timestamps in javascript > apache-arrow@2.0.0 > > See this notebook: > https://observablehq.com/@nite/apache-arrow-timestamp-investigation > > I'm guessing I'm missing something, has anyone got any suggestions, or > decent examples of reading a file created in pandas? I've seen in examples > of apache-arrow@0.3.1 where dates stored as an array of 2 ints. > > File was created with: > > import pandas as pd > pd.read_parquet('sample.parquet') > df.to_feather('sample-seconds.feather') > > Final Q: I'm assuming this is the best place for this question? Happy to > post elsewhere if there's any other forums, or if this should be a JIRA > ticket? > > Thanks! > Andy > --00000000000039aebd05b6ace648 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
So, I figured out the issue here - I had to remove compres= sion=C2=A0from the pyarrow=C2=A0feather.write_feat= her(compression=3D'uncompressed'). Is there any way to read = a compressed feather file in arrow js?=C2=A0
See the comment under the = first answer here:=C2=A0https://stackoverflow.com/questions/64629670/how-to-write-a-pa= ndas-dataframe-to-arrow-file/64648955#64648955
I couldn't=C2=A0f= ind anything in the arrow docs or notebooks on this - I'm assuming that= 's related to javascript=C2=A0compression=C2=A0libraries being so limit= ed.=C2=A0


On Mon, 14 Dec 2020 at 21:32, Andrew Clancy &= lt;nite@achren.org= > wrote:

Hi all,

I have a simple feather file created via a pa= ndas to_feather with a datetime64[ns]=C2=A0column, and cannot get timestamp= s in javascript apache-arrow@2.0.0=C2=A0

See this notebook:
https://observablehq.com/@nite/apache-arr= ow-timestamp-investigation

I'm guessing I'm missing somet= hing, has anyone got any suggestions, or decent examples of reading a file = created in pandas? I've seen in examples of apache-arrow@0.3.1 where da= tes stored as an array of 2 ints.=C2=A0

File was created with:=C2=A0<= /p>

import pandas as pd
pd.read_parquet('= sample.parquet')
df.to_feather('sample-seconds.feather')

Final Q: I'm assuming this is the best place for this que= stion? Happy=C2=A0to post elsewhere if there's any other forums, or if = this should be a JIRA ticket?

Thanks!
Andy

--00000000000039aebd05b6ace648--