kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "XiaoXiang Yu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
Date Tue, 27 Nov 2018 11:54:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700285#comment-16700285
] 

XiaoXiang Yu edited comment on KYLIN-3695 at 11/27/18 11:53 AM:
----------------------------------------------------------------

Hi [~gauravbrills]

I installed the newest stable hive in my test env, which version is 2.3.4( _Hive DDL doc
says 2.2.0 support double with precision_). I found hive didn't not support double with specific/user-defined precision.

e.g.
 *  [ERROR] create table haha(ha double 10); 
 *  [ERROR] create table haha(ha double Precision 17);
 *  [ERROR] create table haha(ha double(22));
 *  [CORRECT] create table haha(ha double Precision);

{quote}hive> use ss;
 OK
 Time taken: 0.015 seconds
 hive> create table haha(ha double 10);
 FAILED: ParseException line 1:28 extraneous input '10' expecting ) near '<EOF>'
 hive> create table haha(ha double Precision 17);
 FAILED: ParseException line 1:38 extraneous input '17' expecting ) near '<EOF>'
 hive> create table haha(ha double(22));
 MismatchedTokenException(334!=347)
 at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
 at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
 at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
 at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
 at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
 at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333)
 at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:208)
 at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77)
 at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70)
 at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)
 at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
 at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
 at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
 at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
 at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
 at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
 at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
 at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
 at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
 FAILED: ParseException line 1:27 mismatched input '(' expecting ) near 'double' in create
table statement
 hive> create table haha(ha double Precision);
 OK
 Time taken: 0.625 seconds
{quote}
 

*So, as far as I can concerned, Hive didn't support such data type definition. It can not
be resolved by kylin dev.*

 

If you found any mistake, please let me know. And I found below links which maybe helpful:

1. [https://cwiki.apache.org/confluence/display/hive/LanguageManual+DDL#LanguageManualDDL-CreateTableCreate/Drop/TruncateTable]

2. https://issues.apache.org/jira/browse/HIVE-13556


was (Author: hit_lacus):
Hi [~gauravbrills]

I installed the newest stable hive in my test env, which version is 2.3.4.I found hive didn't
not support double with specific/user-defined precision.

e.g.
 *  [ERROR] create table haha(ha double 10); 
 *  [ERROR] create table haha(ha double Precision 17);
 *  [ERROR] create table haha(ha double(22));
 *  [CORRECT] create table haha(ha double Precision);

 

 
{quote}hive> use ss;
OK
Time taken: 0.015 seconds
hive> create table haha(ha double 10);
FAILED: ParseException line 1:28 extraneous input '10' expecting ) near '<EOF>'
hive> create table haha(ha double Precision 17);
FAILED: ParseException line 1:38 extraneous input '17' expecting ) near '<EOF>'
hive> create table haha(ha double(22));
MismatchedTokenException(334!=347)
at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:208)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: ParseException line 1:27 mismatched input '(' expecting ) near 'double' in create
table statement
hive> create table haha(ha double Precision);
OK
Time taken: 0.625 seconds
{quote}
 

And I found below links:

1. https://cwiki.apache.org/confluence/display/hive/LanguageManual+DDL#LanguageManualDDL-CreateTableCreate/Drop/TruncateTable

2. https://issues.apache.org/jira/browse/HIVE-13556

> Error while creating hive table through Kylin build cube with mysql imported tables
> -----------------------------------------------------------------------------------
>
>                 Key: KYLIN-3695
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3695
>             Project: Kylin
>          Issue Type: Bug
>          Components: Integration
>    Affects Versions: v2.5.0
>            Reporter: Gaurav Rawat
>            Assignee: XiaoXiang Yu
>            Priority: Minor
>
> HI I am trying to build a cube with Kylin, the data gets souced fine from sqoop but the
next step for creating hive tables fails . Looking at the command being fired it looks weird
as the create statement looks good to me .
> I think the issue is with DOUBLE types as when I remove the same the create statement
works fine . Can someone please help .
> I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
> The errors logs with commands as as below, the table is a msql table which had columns
with DOUBLE type
> Command
> {code:java}
> hive -e "USE default; DROP TABLE IF EXISTS kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368;
CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368
( HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string ,STOCK_STOCK_ID string
,STOCK_DOMICILE string ,STOCK_STOCK_NM string ,STOCK_APPROACH string ,STOCK_STOCK_TYP string
,INVESTOR_ID string ,INVESTOR_NM string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22) ,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22)
) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE LOCATION 's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
ALTER TABLE kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET TBLPROPERTIES('auto.purge'='true');
" --hiveconf hive.merge.mapredfiles=false --hiveconf hive.auto.convert.join=true --hiveconf
dfs.replication=2 --hiveconf hive.exec.compress.output=true --hiveconf hive.auto.convert.join.noconditionaltask=true
--hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false --hiveconf
hive.auto.convert.join.noconditionaltask.size=100000000 --hiveconf hive.stats.autogather=true{code}
> Error is as below
> {code:java}
> OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds MismatchedTokenException(334!=347)
at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617) at
org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808) at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204)
at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77) at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at
java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136) FAILED: ParseException line 15:42 mismatched
input '(' expecting ) near 'double' in create table statement{code}
> [hadoop|https://stackoverflow.com/questions/tagged/hadoop] [hive|https://stackoverflow.com/questions/tagged/hive] [kylin|https://stackoverflow.com/questions/tagged/kylin]
>  
> More details here  [https://stackoverflow.com/questions/53377623/error-while-creating-hive-table-through-kylin-build-cube] .
Though the same got solved when I used DECIMAL type with precision .
>  
> Also observer that if you use DECIMAL(10) it does not work unless it has a precision
while building mode for the same it gives error as tries to use precision as undefined ie
. DECIMAL(10,undefined)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message