hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (MAPREDUCE-7033) Map outputs implicitly rely on permissive umask for shuffle
Date Thu, 25 Jan 2018 22:22:00 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-7033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jason Lowe reassigned MAPREDUCE-7033:

            Assignee: Jason Lowe
            Priority: Critical  (was: Major)
    Target Version/s: 3.1.0, 3.0.1

Raising the priority of this since I recently discovered this long-lingering bug can actually
prevent proper shuffling on secure clusters in Hadoop 3.x.

HADOOP-11347 fixed a bug in the local filesystem so it started honoring the configured fs.permissions.umask-mode
for local files.  That fix now exposes the lack of explicit permissions in the MapReduce framework
for any case where the configured umask is at least 0044.

> Map outputs implicitly rely on permissive umask for shuffle
> -----------------------------------------------------------
>                 Key: MAPREDUCE-7033
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7033
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Critical
> Map tasks do not explicitly set the permissions of their output files for shuffle.  In
a secure cluster the shuffle service is running as a different user than the map task, so
the output files require group readability in order to serve up the data during the shuffle
phase.  If the user's UNIX umask is too restrictive (e.g.: 077) then the map task's file.out
and file.out.index permissions can be too restrictive to allow the shuffle handler to access

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org

View raw message