hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruslan Dautkhanov (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HIVE-14989) FIELDS TERMINATED BY parsing broken when delimiter is more than 1 byte
Date Tue, 18 Oct 2016 04:47:58 GMT

     [ https://issues.apache.org/jira/browse/HIVE-14989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ruslan Dautkhanov resolved HIVE-14989.
--------------------------------------
       Resolution: Duplicate
    Fix Version/s: 0.14.1

> FIELDS TERMINATED BY parsing broken when delimiter is more than 1 byte
> ----------------------------------------------------------------------
>
>                 Key: HIVE-14989
>                 URL: https://issues.apache.org/jira/browse/HIVE-14989
>             Project: Hive
>          Issue Type: Bug
>          Components: File Formats, Parser, Reader
>    Affects Versions: 0.13.0, 0.13.1
>            Reporter: Ruslan Dautkhanov
>             Fix For: 0.14.1
>
>
> FIELDS TERMINATED BY parsing broken when delimiter is more than 1 byte. Delimiter starting
from 2nd character becomes part of returned data. No parsed properly.
> Test case:
> {noformat}
> CREATE external TABLE test_muldelim
> (  string1 STRING,
>    string2 STRING,
>    string3 STRING
> )
>  ROW FORMAT 
>        DELIMITED FIELDS TERMINATED BY '<>'
>       LINES TERMINATED BY '\n'
>  STORED AS TEXTFILE
>   location '/user/hive/test_muldelim'
> {noformat}
> Create a text file under /user/hive/test_muldelim with following 2 lines:
> {noformat}
> data1<>data2<>data3
> aa<>bb<>cc
> {noformat}
> Now notice that two-character delimiter wasn't parsed properly:
> {noformat}
> jdbc:hive2://host.domain.com:1> select * from ruslan_test.test_muldelim ;
> +------------------------+------------------------+------------------------+--+
> | test_muldelim.string1  | test_muldelim.string2  | test_muldelim.string3  |
> +------------------------+------------------------+------------------------+--+
> | data1                  | >data2                 | >data3                 |
> | aa                     | >bb                    | >cc                    |
> +------------------------+------------------------+------------------------+--+
> 2 rows selected (0.453 seconds)
> {noformat}
> The second delimiter's character ('>') became part of the columns to the right (`string2`
and `string3`).
> Table DDL:
> {noformat}
> 0: jdbc:hive2://host.domain.com:1> show create table dafault.test_muldelim ;
> +-----------------------------------------------------------------+--+
> |                         createtab_stmt                          |
> +-----------------------------------------------------------------+--+
> | CREATE EXTERNAL TABLE `default.test_muldelim`(              |
> |   `string1` string,                                             |
> |   `string2` string,                                             |
> |   `string3` string)                                             |
> | ROW FORMAT DELIMITED                                            |
> |   FIELDS TERMINATED BY '<>'                                     |
> |   LINES TERMINATED BY '\n'                                      |
> | STORED AS INPUTFORMAT                                           |
> |   'org.apache.hadoop.mapred.TextInputFormat'                    |
> | OUTPUTFORMAT                                                    |
> |   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'  |
> | LOCATION                                                        |
> |   'hdfs://epsdatalake/user/hive/test_muldelim'              |
> | TBLPROPERTIES (                                                 |
> |   'transient_lastDdlTime'='1476727100')                         |
> +-----------------------------------------------------------------+--+
> 15 rows selected (0.286 seconds)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message