hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6198) ORC file and struct column names are case sensitive
Date Wed, 08 Oct 2014 17:07:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163786#comment-14163786
] 

Hive QA commented on HIVE-6198:
-------------------------------



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12673317/HIVE-6198.3.patch.txt

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1158/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1158/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1158/

Messages:
{noformat}
**** This message was trimmed, see log for full details ****
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_CASE" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_NOT KW_FALSE" using multiple alternatives: 1,
2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_DATE StringLiteral" using multiple alternatives:
1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_NOT KW_TRUE" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_NOT KW_MAP" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_MAP" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_MAP" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_UNIONTYPE" using multiple alternatives:
1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_STRUCT" using multiple alternatives: 1,
2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_NOT KW_IF" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN KW_CASE KW_IF" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:68:4: 
Decision can match input such as "LPAREN LPAREN KW_IF" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:115:5: 
Decision can match input such as "KW_CLUSTER KW_BY LPAREN" using multiple alternatives: 1,
2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:127:5: 
Decision can match input such as "KW_PARTITION KW_BY LPAREN" using multiple alternatives:
1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:138:5: 
Decision can match input such as "KW_DISTRIBUTE KW_BY LPAREN" using multiple alternatives:
1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:149:5: 
Decision can match input such as "KW_SORT KW_BY LPAREN" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:166:7: 
Decision can match input such as "STAR" using multiple alternatives: 1, 2

As a result, alternative(s) 2 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_STRUCT" using multiple alternatives: 4, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_ARRAY" using multiple alternatives: 2, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:179:5: 
Decision can match input such as "KW_UNIONTYPE" using multiple alternatives: 5, 6

As a result, alternative(s) 6 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_TRUE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_NULL" using multiple alternatives: 1, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_FALSE" using multiple alternatives: 3, 8

As a result, alternative(s) 8 were disabled for that input
warning(200): IdentifiersParser.g:261:5: 
Decision can match input such as "KW_DATE StringLiteral" using multiple alternatives: 2, 3

As a result, alternative(s) 3 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "KW_BETWEEN KW_MAP LPAREN" using multiple alternatives: 8,
9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_INTO" using
multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_UNION KW_ALL" using multiple
alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_SORT KW_BY" using multiple
alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_CLUSTER KW_BY" using multiple
alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_LATERAL KW_VIEW" using
multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_MAP LPAREN" using multiple
alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_DISTRIBUTE KW_BY" using
multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_ORDER KW_BY" using multiple
alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_GROUP KW_BY" using multiple
alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:393:5: 
Decision can match input such as "{KW_LIKE, KW_REGEXP, KW_RLIKE} KW_INSERT KW_OVERWRITE" using
multiple alternatives: 2, 9

As a result, alternative(s) 9 were disabled for that input
warning(200): IdentifiersParser.g:518:5: 
Decision can match input such as "{AMPERSAND..BITWISEXOR, DIV..DIVIDE, EQUAL..EQUAL_NS, GREATERTHAN..GREATERTHANOREQUALTO,
KW_AND, KW_ARRAY, KW_BETWEEN..KW_BOOLEAN, KW_CASE, KW_DOUBLE, KW_FLOAT, KW_IF, KW_IN, KW_INT,
KW_LIKE, KW_MAP, KW_NOT, KW_OR, KW_REGEXP, KW_RLIKE, KW_SMALLINT, KW_STRING..KW_STRUCT, KW_TINYINT,
KW_UNIONTYPE, KW_WHEN, LESSTHAN..LESSTHANOREQUALTO, MINUS..NOTEQUAL, PLUS, STAR, TILDE}" using
multiple alternatives: 1, 3

As a result, alternative(s) 3 were disabled for that input
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec ---
Downloading: http://www.datanucleus.org/downloads/maven2/net/hydromatic/linq4j/0.4/linq4j-0.4.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Hive .............................................. SUCCESS [13.166s]
[INFO] Hive Shims Common ................................. SUCCESS [8.514s]
[INFO] Hive Shims 0.20 ................................... SUCCESS [3.916s]
[INFO] Hive Shims Secure Common .......................... SUCCESS [4.914s]
[INFO] Hive Shims 0.20S .................................. SUCCESS [2.273s]
[INFO] Hive Shims 0.23 ................................... SUCCESS [7.673s]
[INFO] Hive Shims ........................................ SUCCESS [0.808s]
[INFO] Hive Common ....................................... SUCCESS [9.435s]
[INFO] Hive Serde ........................................ SUCCESS [14.651s]
[INFO] Hive Metastore .................................... SUCCESS [36.421s]
[INFO] Hive Ant Utilities ................................ SUCCESS [1.741s]
[INFO] Hive Query Language ............................... FAILURE [33.011s]
[INFO] Hive Service ...................................... SKIPPED
[INFO] Hive Accumulo Handler ............................. SKIPPED
[INFO] Hive JDBC ......................................... SKIPPED
[INFO] Hive Beeline ...................................... SKIPPED
[INFO] Hive CLI .......................................... SKIPPED
[INFO] Hive Contrib ...................................... SKIPPED
[INFO] Hive HBase Handler ................................ SKIPPED
[INFO] Hive HCatalog ..................................... SKIPPED
[INFO] Hive HCatalog Core ................................ SKIPPED
[INFO] Hive HCatalog Pig Adapter ......................... SKIPPED
[INFO] Hive HCatalog Server Extensions ................... SKIPPED
[INFO] Hive HCatalog Webhcat Java Client ................. SKIPPED
[INFO] Hive HCatalog Webhcat ............................. SKIPPED
[INFO] Hive HCatalog Streaming ........................... SKIPPED
[INFO] Hive HWI .......................................... SKIPPED
[INFO] Hive ODBC ......................................... SKIPPED
[INFO] Hive Shims Aggregator ............................. SKIPPED
[INFO] Hive TestUtils .................................... SKIPPED
[INFO] Hive Packaging .................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2:19.160s
[INFO] Finished at: Wed Oct 08 13:06:56 EDT 2014
[INFO] Final Memory: 68M/463M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(default) on project hive-exec: Error resolving project artifact: Could not transfer artifact
net.hydromatic:linq4j:pom:0.4 from/to datanucleus (http://www.datanucleus.org/downloads/maven2):
Access denied to: http://www.datanucleus.org/downloads/maven2/net/hydromatic/linq4j/0.4/linq4j-0.4.pom,
ReasonPhrase: Forbidden. for project net.hydromatic:linq4j:jar:0.4 -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following
articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hive-exec
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12673317

> ORC file and struct column names are case sensitive
> ---------------------------------------------------
>
>                 Key: HIVE-6198
>                 URL: https://issues.apache.org/jira/browse/HIVE-6198
>             Project: Hive
>          Issue Type: Bug
>          Components: CLI, File Formats
>    Affects Versions: 0.11.0, 0.12.0
>            Reporter: Viraj Bhat
>            Assignee: Navis
>         Attachments: HIVE-6198.1.patch.txt, HIVE-6198.2.patch.txt, HIVE-6198.3.patch.txt
>
>
> HiveQL document states that the "Table names and column names are case insensitive".
But the struct behavior for ORC file is different. 
> Consider a sample text file:
> {code}
> $ cat data.txt
> line1|key11:value11,key12:value12,key13:value13|a,b,c|one,two
> line2|key21:value21,key22:value22,key23:value23|d,e,f|three,four
> line3|key31:value31,key32:value32,key33:value33|g,h,i|five,six
> {code}
> Creating a table stored as txt and then using this to create a table stored as orc 
> {code}
> CREATE TABLE orig (
>   str STRING,
>   mp  MAP<STRING,STRING>,
>   lst ARRAY<STRING>,
>   strct STRUCT<A:STRING,B:STRING>
> ) ROW FORMAT DELIMITED
>     FIELDS TERMINATED BY '|'
>     COLLECTION ITEMS TERMINATED BY ','
>     MAP KEYS TERMINATED BY ':';
> LOAD DATA LOCAL INPATH 'data.txt' INTO TABLE orig;
> CREATE TABLE tableorc (
>   str STRING,
>   mp  MAP<STRING,STRING>,
>   lst ARRAY<STRING>,
>   strct STRUCT<A:STRING,B:STRING>
> ) STORED AS ORC;
> INSERT OVERWRITE TABLE tableorc SELECT * FROM orig;
> {code}
> Suppose we project columns or read the *strct* columns for both table types, here are
the results. I have also tested the same with *RC*. The behavior is similar to *txt* files.
> {code}
> hive> SELECT * FROM orig;
> line1   {"key11":"value11","key12":"value12","key13":"value13"} ["a","b","c"]  
> {"a":"one","b":"two"}
> line2   {"key21":"value21","key22":"value22","key23":"value23"} ["d","e","f"]  
> {"a":"three","b":"four"}
> line3   {"key31":"value31","key32":"value32","key33":"value33"} ["g","h","i"]  
> {"a":"five","b":"six"}
> Time taken: 0.126 seconds, Fetched: 3 row(s)
> hive> SELECT * FROM tableorc;
> line1   {"key12":"value12","key11":"value11","key13":"value13"} ["a","b","c"]  
> {"A":"one","B":"two"}
> line2   {"key21":"value21","key23":"value23","key22":"value22"} ["d","e","f"]  
> {"A":"three","B":"four"}
> line3   {"key33":"value33","key31":"value31","key32":"value32"} ["g","h","i"]  
> {"A":"five","B":"six"}
> Time taken: 0.178 seconds, Fetched: 3 row(s)
> hive> SELECT strct FROM tableorc;
> {"a":"one","b":"two"}
> {"a":"three","b":"four"}
> {"a":"five","b":"six"}
> hive>SELECT strct.A FROM orig;
> one
> three
> five
> hive>SELECT strct.a FROM orig;
> one
> three
> five
> hive>SELECT strct.A FROM tableorc;
> one
> three
> five
> hive>SELECT strct.a FROM tableorc;
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched: 
> Job 0: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
> {code}
> So it seems that ORC behaves differently for struct columns. Also why are we storing
the column names for struct for the other types as CASE SENSITIVE? What is the standard for
Hive QL with respect to structs?
> Regards
> Viraj



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message