hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tarek Abouzeid (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (MAPREDUCE-7033) Map outputs implicitly rely on permissive umask for shuffle
Date Fri, 17 May 2019 07:48:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841977#comment-16841977
] 

Tarek Abouzeid edited comment on MAPREDUCE-7033 at 5/17/19 7:47 AM:
--------------------------------------------------------------------

Hello, 

i have an issue, the file.out and file.out.index are being created with different permissions, 

 
{code:java}
-rw-------. 1 hive hadoop 28 May 16 16:17 file.out
-rw-r-----. 1 hive hadoop 32 May 16 16:17 file.out.index
{code}
for parent directories
{code:java}
drwx--x---. 3 hive hadoop 16 May 16 16:16 filecache
drwxr-s---. 3 hive hadoop 60 May 16 16:16 output
{code}
which leads to being unable to read the result of the map task, i am using hortonworks HDP
3.1.0.0-78 , is there any configuration that i can set to enforce the correct permissions
? 

 


was (Author: tarekabouzeid91):
HHello, 

i have an issue, the file.out and file.out.index are being created with different permissions, 

 
{code:java}
-rw-------. 1 hive hadoop 28 May 16 16:17 file.out
-rw-r-----. 1 hive hadoop 32 May 16 16:17 file.out.index
{code}
for parent directories
{code:java}
drwx--x---. 3 hive hadoop 16 May 16 16:16 filecache
drwxr-s---. 3 hive hadoop 60 May 16 16:16 output
{code}
which leads to being unable to read the result of the map task, i am using hortonworks HDP
3.1.0.0-78 , is there any configuration that i can set to enforce the correct permissions
? 

 

> Map outputs implicitly rely on permissive umask for shuffle
> -----------------------------------------------------------
>
>                 Key: MAPREDUCE-7033
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7033
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Critical
>             Fix For: 3.1.0, 3.0.1
>
>         Attachments: MAPREDUCE-7033.001.patch, MAPREDUCE-7033.002.patch
>
>
> Map tasks do not explicitly set the permissions of their output files for shuffle.  In
a secure cluster the shuffle service is running as a different user than the map task, so
the output files require group readability in order to serve up the data during the shuffle
phase.  If the user's UNIX umask is too restrictive (e.g.: 077) then the map task's file.out
and file.out.index permissions can be too restrictive to allow the shuffle handler to access
them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message