hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Dere (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-7774) Issues with location path for temporary external tables
Date Wed, 20 Aug 2014 20:24:26 GMT

     [ https://issues.apache.org/jira/browse/HIVE-7774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jason Dere updated HIVE-7774:
-----------------------------

       Resolution: Fixed
    Fix Version/s: 0.14.0
           Status: Resolved  (was: Patch Available)

Committed to trunk, thanks for review Ashutosh.

> Issues with location path for temporary external tables
> -------------------------------------------------------
>
>                 Key: HIVE-7774
>                 URL: https://issues.apache.org/jira/browse/HIVE-7774
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Jason Dere
>            Assignee: Jason Dere
>             Fix For: 0.14.0
>
>         Attachments: HIVE-7774.1.patch
>
>
> Depending on the location string passed into temp external table, a query requiring a
map/reduce job will fail.  Example:
> {noformat}
> create temporary external table tmp1 (c1 string) location '/tmp/tmp1';
> describe extended tmp1;
> select count(*) from tmp1;
> {noformat}
> Will result in the following error:
> {noformat}
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: Error in configuring object
> 	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> 	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> 	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> 	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> 	... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> 	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
> 	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
> 	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> 	at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
> 	... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
> 	... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> 	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:154)
> 	... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException:
Configuration and input path are inconsistent
> 	at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:404)
> 	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:123)
> 	... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Configuration and input
path are inconsistent
> 	at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:398)
> 	... 23 more
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> {noformat}
> If the location is set to 'hdfs:/tmp/tmp1', it gets the following error:
> {noformat}
> java.io.IOException: cannot find dir = hdfs://node-1.example.com:8020/tmp/tmp1/tmp1.txt
in pathToPartitionInfo: [hdfs:/tmp/tmp1]
> 	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:344)
> 	at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:306)
> 	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat$CombineHiveInputSplit.<init>(CombineHiveInputFormat.java:108)
> 	at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:455)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
> 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
> 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> 	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
> 	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
> 	at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:420)
> 	at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:161)
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> 	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1555)
> 	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1322)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1136)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:960)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:265)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:217)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:427)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:363)
> 	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:460)
> 	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:476)
> 	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:694)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Job Submission failed with exception 'java.io.IOException(cannot find dir = hdfs://node-1.example.com:8020/tmp/tmp1/tmp1.txt
in pathToPartitionInfo: [hdfs:/tmp/tmp1])'
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> {noformat}
> The only location string value that works properly here is 'hdfs://hostname/tmp/tmp1'.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message