hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Zimmerman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4605) Hive job fails while closing reducer output - Unable to rename
Date Wed, 29 Mar 2017 04:47:42 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946521#comment-15946521
] 

Gabriel Zimmerman commented on HIVE-4605:
-----------------------------------------


I am seeing the same issue and maybe this sheds some light. Using Hive version 1.0.0 and below
(haven't tried later versions) on EMR, this error happens consistently. 

The table uses its own storage handler but when I use "create external table" to create the
table, somehow Hive thinks the table is native, which means that it runs commit on the FS
paths causing the error (see stack trace above). Removing external fixed the issue. Maybe
overwrite in the above case had a similar result?

> Hive job fails while closing reducer output - Unable to rename
> --------------------------------------------------------------
>
>                 Key: HIVE-4605
>                 URL: https://issues.apache.org/jira/browse/HIVE-4605
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.11.0, 0.12.0, 0.13.0, 0.13.1
>         Environment: OS: 2.6.18-194.el5xen #1 SMP Fri Apr 2 15:34:40 EDT 2010 x86_64
x86_64 x86_64 GNU/Linux
> Hadoop 1.1.2
>            Reporter: Link Qian
>            Assignee: Brock Noland
>         Attachments: HIVE-4605.patch
>
>
> 1, create a table with ORC storage model
> create table iparea_analysis_orc (network int, ip string,  .... )
> stored as ORC;
> 2, insert table iparea_analysis_orc select  network, ip, .... , the script success, but
failed after add *OVERWRITE* keyword.  the main error log list as here.
> ava.lang.RuntimeException: Hive Runtime Error while closing operators: Unable to rename
output from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-10000/_tmp.000000_0
to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-10000/000000_0
> 	at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:317)
> 	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:530)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
> 	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output
from: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_task_tmp.-ext-10000/_tmp.000000_0
to: hdfs://qa3hop001.uucun.com:9000/tmp/hive-hadoop/hive_2013-05-24_15-11-06_511_7746839019590922068/_tmp.-ext-10000/000000_0
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:197)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$300(FileSinkOperator.java:108)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:867)
> 	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:588)
> 	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> 	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> 	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:597)
> 	at org.apache.hadoop.hive.ql.exec.ExecReducer.close(ExecReducer.java:309)
> 	... 7 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message