hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Pivovarov <apivova...@gmail.com>
Subject query uses WITH blocks and throws exception if run as Oozie hive action (hive-0.13.1)
Date Fri, 15 May 2015 21:26:28 GMT
Hi Everyone

I'm using hive-0.13.1   (HDP-2.1.5) and getting the following stacktrace if
run my query (which has WITH block) via Oozie.   (BTW, the query works fine
in CLI)

I can't put exact query but the structure is similar to

create table my_consumer
as
with sacusaloan as (select distinct e,f,g from E)

select A.a, A.b, A.c,
  if(sacusaloan.id is null, 0, 1) as sacusaloan_status
from (select a,b,c from A) A
left join sacusaloan on (...)

8799 [main] INFO  hive.ql.parse.ParseDriver  - Parse Completed
8799 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  -
</PERFLOG method=parse start=1431723485500 end=1431723485602
duration=102 from=org.apache.hadoop.hive.ql.Driver>
8799 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG
method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
8834 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Starting Semantic Analysis
8837 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Creating table wk_qualified_outsource_loan_consumer position=13
8861 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Completed phase 1 of Semantic Analysis
8861 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for source tables
8865 [main] ERROR hive.ql.metadata.Hive  -
NoSuchObjectException(message:default.sacusaloan table not found)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29338)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29306)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result.read(ThriftHiveMetastore.java:29237)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1036)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1022)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:997)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
	at com.sun.proxy.$Proxy18.getTable(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:976)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:918)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1263)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1232)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9252)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:359)
	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:456)
	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:466)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:749)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
	at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:316)
	at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:277)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
	at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

8872 [main] ERROR hive.ql.metadata.Hive  -
NoSuchObjectException(message:default.recallloan table not found)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29338)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29306)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result.read(ThriftHiveMetastore.java:29237)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1036)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1022)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:997)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
	at com.sun.proxy.$Proxy18.getTable(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:976)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:918)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1263)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1232)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9252)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:359)
	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:456)
	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:466)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:749)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
	at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:316)
	at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:277)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
	at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

9700 [main] ERROR hive.ql.metadata.Hive  -
NoSuchObjectException(message:default.loanmob table not found)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29338)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:29306)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result.read(ThriftHiveMetastore.java:29237)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1036)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1022)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:997)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
	at com.sun.proxy.$Proxy18.getTable(Unknown Source)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:976)
	at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:918)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1263)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1232)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9252)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:427)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:323)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:980)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1045)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:916)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:906)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:359)
	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:456)
	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:466)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:749)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
	at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:316)
	at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:277)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
	at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:66)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)

9708 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for subqueries
9708 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for source tables
9798 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for subqueries
9798 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for destination tables
9798 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for source tables
9815 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for subqueries
9815 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for destination tables
9815 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for source tables
9815 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for subqueries
9815 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for source tables
9827 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for subqueries
9827 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for destination tables
9827 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for destination tables
9827 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for source tables
9852 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for subqueries
9852 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for destination tables
9852 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for destination tables
9876 [main] INFO  org.apache.hadoop.hive.ql.exec.Utilities  - Create
dirs hdfs://hadev/tmp/hive-svc-yarn/hive_2015-05-15_13-58-05_500_5122268870471366216-1
with permission rwxrwxrwx recursive false
9894 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Completed getting MetaData in Semantic Analysis
10277 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for source tables
10289 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for subqueries
10290 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for destination tables
10294 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for source tables
10294 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for subqueries
10294 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for source tables
10320 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for subqueries
10321 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for destination tables
10321 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Get metadata for destination tables
10816 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  -
Set stats collection dir :
hdfs://hadev/tmp/hive-svc-yarn/hive_2015-05-15_13-58-05_500_5122268870471366216-1/-ext-10002

Mime
View raw message