hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harshit Mathur <mathursh...@gmail.com>
Subject Re: File permission of Hadoop staging directory getting changed time to time
Date Mon, 06 Apr 2015 09:29:38 GMT
try running with sudo hive.

On Mon, Apr 6, 2015 at 1:40 PM, Inosh Goonewardena <inoshmrt@gmail.com>
wrote:

> Hi All,
>
> Really appreciate any input on this issue.
>
> Thanks and Regards,
> Inosh
>
> On Tue, Jan 20, 2015 at 10:29 PM, Inosh Goonewardena <inoshmrt@gmail.com>
> wrote:
>
>> Hi All,
>>
>> We have setup a Hadoop cluster using hadoop 1.0.4 and we use Hive to
>> submit map/reduce jobs. But we have noticed that these map/reduce job
>> submission failing time to time due to a permission issue occurring in
>> Hadoop. When this happens, we have to use the hadoop hdfs command and
>> change the file permission of mapreduce staging directory. Following is the
>> error that is getting logged.
>>
>> [2015-01-16 05:30:13,314] ERROR {org.apache.hadoop.hive.ql.exec.Task} -
>> Job Submission failed with exception 'java.io.IOException(The
>> ownership/permissions on the staging directory hdfs://
>> hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging is
>> not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------)'
>> java.io.IOException: The ownership/permissions on the staging directory
>> hdfs://hadoop0.test.com:9000/mnt/hadoop_tmp/mapred/staging/inosh/.staging
>> is not as expected. It is owned by inosh and permissions are rwxr-xr-x. The
>> directory must be owned by the submitter inosh or by inosh and permissions
>> must be rwx------
>> at
>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:108)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:798)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:792)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1123)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:792)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:766)
>> at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:460)
>> at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
>> at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
>> at
>> org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
>> at
>> org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
>> ...
>>
>>
>> When we get this error we go to hadoop name node and execute following
>> command to set proper permission to the folder mentioned in the exception.
>> ./hadoop dfs -chmod 700 /mnt/hadoop_tmp/mapred/staging/inosh/.staging
>>
>>
>> We would like know why this happens and how to fix this permanently.
>>
>>
>> Thanks and Regards,
>> Inosh
>>
>
>


-- 
Harshit Mathur

Mime
View raw message