drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arina Ielchiieva (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (DRILL-3688) Drill should honor "skip.header.line.count" and "skip.footer.line.count" attributes of Hive table
Date Thu, 18 Feb 2016 14:03:18 GMT

     [ https://issues.apache.org/jira/browse/DRILL-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arina Ielchiieva updated DRILL-3688:
------------------------------------
    Description: 
Currently Drill does not honor the "skip.header.line.count" attribute of Hive table.
It may cause some other format conversion issue.

Reproduce:

1. Create a Hive table
{code}
create table h1db.testheader(col0 string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
STORED AS TEXTFILE
tblproperties("skip.header.line.count"="1");
{code}
2. Prepare a sample data:
{code}
# cat test.data
col0
2015-01-01
{code}
3. Load sample data into Hive
{code}
LOAD DATA LOCAL INPATH '/xxx/test.data' OVERWRITE INTO TABLE h1db.testheader;
{code}
4. Hive
{code}
hive> select * from h1db.testheader ;
OK
2015-01-01
Time taken: 0.254 seconds, Fetched: 1 row(s)
{code}
5. Drill
{code}
>  select * from hive.h1db.testheader ;
+-------------+
|    col0     |
+-------------+
| col0        |
| 2015-01-01  |
+-------------+
2 rows selected (0.257 seconds)

> select cast(col0 as date) from hive.h1db.testheader ;
Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must be in the range
[1,12]

Fragment 0:0

[Error Id: 34353702-ca27-440b-a4f4-0c9f79fc8ccd on h1.poc.com:31010]

  (org.joda.time.IllegalFieldValueException) Value 0 for monthOfYear must be in the range
[1,12]
    org.joda.time.field.FieldUtils.verifyValueBounds():236
    org.joda.time.chrono.BasicChronology.getDateMidnightMillis():613
    org.joda.time.chrono.BasicChronology.getDateTimeMillis():159
    org.joda.time.chrono.AssembledChronology.getDateTimeMillis():120
    org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.memGetDate():261
    org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getDate():218
    org.apache.drill.exec.test.generated.ProjectorGen0.doEval():67
    org.apache.drill.exec.test.generated.ProjectorGen0.projectRecords():62
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():172
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
    org.apache.drill.exec.record.AbstractRecordBatch.next():147
    org.apache.drill.exec.physical.impl.BaseRootExec.next():83
    org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():79
    org.apache.drill.exec.physical.impl.BaseRootExec.next():73
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():261
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():255
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1566
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():255
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1142
    java.util.concurrent.ThreadPoolExecutor$Worker.run():617
    java.lang.Thread.run():745 (state=,code=0)
{code}

Also "skip.footer.line.count" should be taken into account.
If "skip.header.line.count" or "skip.footer.line.count" has incorrect value in Hive, throw
appropriate exception in Drill.
Ex: Hive table property skip.header.line.count value 'someValue' is non-numeric

  was:
Currently Drill does not honor the "skip.header.line.count" attribute of Hive table.
It may cause some other format conversion issue.

Reproduce:

1. Create a Hive table
{code}
create table h1db.testheader(col0 string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
STORED AS TEXTFILE
tblproperties("skip.header.line.count"="1");
{code}
2. Prepare a sample data:
{code}
# cat test.data
col0
2015-01-01
{code}
3. Load sample data into Hive
{code}
LOAD DATA LOCAL INPATH '/xxx/test.data' OVERWRITE INTO TABLE h1db.testheader;
{code}
4. Hive
{code}
hive> select * from h1db.testheader ;
OK
2015-01-01
Time taken: 0.254 seconds, Fetched: 1 row(s)
{code}
5. Drill
{code}
>  select * from hive.h1db.testheader ;
+-------------+
|    col0     |
+-------------+
| col0        |
| 2015-01-01  |
+-------------+
2 rows selected (0.257 seconds)

> select cast(col0 as date) from hive.h1db.testheader ;
Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must be in the range
[1,12]

Fragment 0:0

[Error Id: 34353702-ca27-440b-a4f4-0c9f79fc8ccd on h1.poc.com:31010]

  (org.joda.time.IllegalFieldValueException) Value 0 for monthOfYear must be in the range
[1,12]
    org.joda.time.field.FieldUtils.verifyValueBounds():236
    org.joda.time.chrono.BasicChronology.getDateMidnightMillis():613
    org.joda.time.chrono.BasicChronology.getDateTimeMillis():159
    org.joda.time.chrono.AssembledChronology.getDateTimeMillis():120
    org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.memGetDate():261
    org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getDate():218
    org.apache.drill.exec.test.generated.ProjectorGen0.doEval():67
    org.apache.drill.exec.test.generated.ProjectorGen0.projectRecords():62
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():172
    org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93
    org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
    org.apache.drill.exec.record.AbstractRecordBatch.next():147
    org.apache.drill.exec.physical.impl.BaseRootExec.next():83
    org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():79
    org.apache.drill.exec.physical.impl.BaseRootExec.next():73
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():261
    org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():255
    java.security.AccessController.doPrivileged():-2
    javax.security.auth.Subject.doAs():422
    org.apache.hadoop.security.UserGroupInformation.doAs():1566
    org.apache.drill.exec.work.fragment.FragmentExecutor.run():255
    org.apache.drill.common.SelfCleaningRunnable.run():38
    java.util.concurrent.ThreadPoolExecutor.runWorker():1142
    java.util.concurrent.ThreadPoolExecutor$Worker.run():617
    java.lang.Thread.run():745 (state=,code=0)
{code}

        Summary: Drill should honor "skip.header.line.count" and "skip.footer.line.count"
attributes of Hive table  (was: Drill should honor "skip.header.line.count" attribute of Hive
table)

> Drill should honor "skip.header.line.count" and "skip.footer.line.count" attributes of
Hive table
> -------------------------------------------------------------------------------------------------
>
>                 Key: DRILL-3688
>                 URL: https://issues.apache.org/jira/browse/DRILL-3688
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - Hive
>    Affects Versions: 1.1.0
>         Environment: 1.1
>            Reporter: Hao Zhu
>            Assignee: Arina Ielchiieva
>             Fix For: Future
>
>
> Currently Drill does not honor the "skip.header.line.count" attribute of Hive table.
> It may cause some other format conversion issue.
> Reproduce:
> 1. Create a Hive table
> {code}
> create table h1db.testheader(col0 string)
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '|'
> STORED AS TEXTFILE
> tblproperties("skip.header.line.count"="1");
> {code}
> 2. Prepare a sample data:
> {code}
> # cat test.data
> col0
> 2015-01-01
> {code}
> 3. Load sample data into Hive
> {code}
> LOAD DATA LOCAL INPATH '/xxx/test.data' OVERWRITE INTO TABLE h1db.testheader;
> {code}
> 4. Hive
> {code}
> hive> select * from h1db.testheader ;
> OK
> 2015-01-01
> Time taken: 0.254 seconds, Fetched: 1 row(s)
> {code}
> 5. Drill
> {code}
> >  select * from hive.h1db.testheader ;
> +-------------+
> |    col0     |
> +-------------+
> | col0        |
> | 2015-01-01  |
> +-------------+
> 2 rows selected (0.257 seconds)
> > select cast(col0 as date) from hive.h1db.testheader ;
> Error: SYSTEM ERROR: IllegalFieldValueException: Value 0 for monthOfYear must be in the
range [1,12]
> Fragment 0:0
> [Error Id: 34353702-ca27-440b-a4f4-0c9f79fc8ccd on h1.poc.com:31010]
>   (org.joda.time.IllegalFieldValueException) Value 0 for monthOfYear must be in the range
[1,12]
>     org.joda.time.field.FieldUtils.verifyValueBounds():236
>     org.joda.time.chrono.BasicChronology.getDateMidnightMillis():613
>     org.joda.time.chrono.BasicChronology.getDateTimeMillis():159
>     org.joda.time.chrono.AssembledChronology.getDateTimeMillis():120
>     org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.memGetDate():261
>     org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.getDate():218
>     org.apache.drill.exec.test.generated.ProjectorGen0.doEval():67
>     org.apache.drill.exec.test.generated.ProjectorGen0.projectRecords():62
>     org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork():172
>     org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext():93
>     org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():129
>     org.apache.drill.exec.record.AbstractRecordBatch.next():147
>     org.apache.drill.exec.physical.impl.BaseRootExec.next():83
>     org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():79
>     org.apache.drill.exec.physical.impl.BaseRootExec.next():73
>     org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():261
>     org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():255
>     java.security.AccessController.doPrivileged():-2
>     javax.security.auth.Subject.doAs():422
>     org.apache.hadoop.security.UserGroupInformation.doAs():1566
>     org.apache.drill.exec.work.fragment.FragmentExecutor.run():255
>     org.apache.drill.common.SelfCleaningRunnable.run():38
>     java.util.concurrent.ThreadPoolExecutor.runWorker():1142
>     java.util.concurrent.ThreadPoolExecutor$Worker.run():617
>     java.lang.Thread.run():745 (state=,code=0)
> {code}
> Also "skip.footer.line.count" should be taken into account.
> If "skip.header.line.count" or "skip.footer.line.count" has incorrect value in Hive,
throw appropriate exception in Drill.
> Ex: Hive table property skip.header.line.count value 'someValue' is non-numeric



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message