Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Wed, 5 Oct 2016 10:55:20 +0000 (UTC)
From: "Vitalii Diravka (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.12922180.1450221668000.741297.1475664920458@Atlassian.JIRA>
In-Reply-To: <JIRA.12922180.1450221668000@Atlassian.JIRA>
References: <JIRA.12922180.1450221668000@Atlassian.JIRA> <JIRA.12922180.1450221668394@arcas>
Subject: [jira] [Commented] (DRILL-4203) Parquet File : Date is stored
 wrongly
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 05 Oct 2016 10:55:22 -0000


    [ https://issues.apache.org/jira/browse/DRILL-4203?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1554=
8372#comment-15548372 ]=20

Vitalii Diravka commented on DRILL-4203:
----------------------------------------

[~rkins] If parquet metadata [consists "creator: parquet-mr"|https://github=
.com/vdiravka/drill/blob/501e7a2fc033a4b99c1cc75adcaa4d835a9acb2e/exec/java=
-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetReaderUtilit=
y.java#L192] it is still could be generated with drill. So to determine it =
we [check | https://github.com/vdiravka/drill/blob/501e7a2fc033a4b99c1cc75a=
dcaa4d835a9acb2e/exec/java-exec/src/main/java/org/apache/drill/exec/store/p=
arquet/ParquetReaderUtility.java#L239] column metadata min/max values.

And parquet files stores date values as INT32 that stores the number of day=
s from the Unix epoch [https://github.com/Parquet/parquet-format/blob/maste=
r/LogicalTypes.md#date]. So it is possible to store dates over 10,000 years=
. [Here (DRILL-4763)|https://issues.apache.org/jira/browse/DRILL-4763] the =
example how drill now reads correct parquet files which are generated from =
another tool (-11395-10-18T00:00:00.000-07:52:58 0).

> Parquet File : Date is stored wrongly
> -------------------------------------
>
>                 Key: DRILL-4203
>                 URL: https://issues.apache.org/jira/browse/DRILL-4203
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.4.0
>            Reporter: St=C3=A9phane Trou
>            Assignee: Vitalii Diravka
>            Priority: Critical
>             Fix For: 1.9.0
>
>
> Hello,
> I have some problems when i try to read parquet files produce by drill wi=
th  Spark,  all dates are corrupted.
> I think the problem come from drill :)
> {code}
> cat /tmp/date_parquet.csv=20
> Epoch,1970-01-01
> {code}
> {code}
> 0: jdbc:drill:zk=3Dlocal> select columns[0] as name, cast(columns[1] as d=
ate) as epoch_date from dfs.tmp.`date_parquet.csv`;
> +--------+-------------+
> |  name  | epoch_date  |
> +--------+-------------+
> | Epoch  | 1970-01-01  |
> +--------+-------------+
> {code}
> {code}
> 0: jdbc:drill:zk=3Dlocal> create table dfs.tmp.`buggy_parquet`as select c=
olumns[0] as name, cast(columns[1] as date) as epoch_date from dfs.tmp.`dat=
e_parquet.csv`;
> +-----------+----------------------------+
> | Fragment  | Number of records written  |
> +-----------+----------------------------+
> | 0_0       | 1                          |
> +-----------+----------------------------+
> {code}
> When I read the file with parquet tools, i found =20
> {code}
> java -jar parquet-tools-1.8.1.jar head /tmp/buggy_parquet/
> name =3D Epoch
> epoch_date =3D 4881176
> {code}
> According to [https://github.com/Parquet/parquet-format/blob/master/Logic=
alTypes.md#date], epoch_date should be equals to 0.
> Meta :=20
> {code}
> java -jar parquet-tools-1.8.1.jar meta /tmp/buggy_parquet/
> file:        file:/tmp/buggy_parquet/0_0_0.parquet=20
> creator:     parquet-mr version 1.8.1-drill-r0 (build 6b605a4ea05b66e1a6b=
f843353abcb4834a4ced8)=20
> extra:       drill.version =3D 1.4.0=20
> file schema: root=20
> -------------------------------------------------------------------------=
-------
> name:        OPTIONAL BINARY O:UTF8 R:0 D:1
> epoch_date:  OPTIONAL INT32 O:DATE R:0 D:1
> row group 1: RC:1 TS:93 OFFSET:4=20
> -------------------------------------------------------------------------=
-------
> name:         BINARY SNAPPY DO:0 FPO:4 SZ:52/50/0,96 VC:1 ENC:RLE,BIT_PAC=
KED,PLAIN
> epoch_date:   INT32 SNAPPY DO:0 FPO:56 SZ:45/43/0,96 VC:1 ENC:RLE,BIT_PAC=
KED,PLAIN
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)