hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15571) After HADOOP-13440, multiple filesystems/file-contexts created with the same Configuration object are forced to have the same umask
Date Mon, 02 Jul 2018 17:44:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16530232#comment-16530232
] 

Xiaoyu Yao commented on HADOOP-15571:
-------------------------------------

{quote}That was my first approach too, but there are too many uses of FileContext (in YARN
and potentially as well as in downstream projects) and forcing all of them to create a new
Configuration object is not right. This creation of such a new Config object per URI was not
needed in 2.x.
{quote}
[~vinodkv], that makes sense to me. +1 for adding a regression test for the fix. 

 

> After HADOOP-13440, multiple filesystems/file-contexts created with the same Configuration
object are forced to have the same umask
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-15571
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15571
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>            Priority: Critical
>         Attachments: HADOOP-15571.txt
>
>
> Ran into a super hard-to-debug issue due to this. [Edit: Turns out the same issue as
YARN-5749 that [~Tao Yang] ran into]
> h4. Issue
> Configuration conf = new Configuration();
>  fc1 = FileContext.getFileContext(uri1, conf);
>  fc2 = FileContext.getFileContext(uri2, conf);
>  fc.setUMask(umask_for_fc1); // Screws up umask for fc2 also!
> This was not the case before HADOOP-13440.
> h4. Symptoms:
> h5. Scenario I ran into
> When trying to localize a HDFS directory (hdfs:///my/dir/1.txt), NodeManager tries to
replicate the directory structure on the local file-system ($yarn-local-dirs/filecache/my/dir/1.txt).
> Now depending on whether NM has ever done a log-aggregation (completely unrelated code
that sets umask to be 137 for its own files on HDFS), the directories /my and /my/dir on local-fs
may have different permissions. In the specific case where NM did log-aggregation, /my/dir
was created with 137 umask and so localization of 1.txt completely failed due to absent directory
executable permissions!
> h5. Previous scenarios:
> We ran into this before in test-cases and instead of fixing the root-cause, we just fixed
the test-cases: YARN-5679 / YARN-5749



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message