hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eugene Koifman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15142) CompactorMR fails with FileNotFoundException
Date Tue, 06 Dec 2016 22:52:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726982#comment-15726982
] 

Eugene Koifman commented on HIVE-15142:
---------------------------------------

Which user is you metastore running as?  That is the user submitting the compaction job to
Hadoop not the user running CLI.

Did you configure the the metastore user as a proxy user for "test"?

This what is expected to work.  If it does, then we should try to improve the error msg if
possible since a lot of people get confused by this.



> CompactorMR fails with FileNotFoundException
> --------------------------------------------
>
>                 Key: HIVE-15142
>                 URL: https://issues.apache.org/jira/browse/HIVE-15142
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 1.0.0
>            Reporter: Eugene Koifman
>            Assignee: Wei Zheng
>            Priority: Critical
>
> {noformat}
> No of maps and reduces are 0 job_1478131263487_0009
> Job commit failed: java.io.FileNotFoundException: File hdfs://....../_tmp_80e50691-c9fe-485a-8e1e-ba20331c7d97
does not exist.
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:904)
> at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:113)
> at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:966)
> at org.apache.hadoop.hdfs.DistributedFileSystem$21.doCall(DistributedFileSystem.java:962)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:962)
> at org.apache.hadoop.hive.ql.txn.compactor.CompactorMR$CompactorOutputCommitter.commitJob(CompactorMR.java:776)
> at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
> at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285)
> at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> when Compactor doesn't have permissions to write to the table dir.
> Evidently we loose the exception when creating CompactorMR.TMP_LOCATION and only see
that the file is not there on commit of the job



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message