hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (HADOOP-15571) After HADOOP-13440, multiple filesystems/file-contexts created with the same Configuration object are forced to have the same umask
Date Mon, 02 Jul 2018 04:13:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-15571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vinod Kumar Vavilapalli reassigned HADOOP-15571:
------------------------------------------------

    Assignee: Vinod Kumar Vavilapalli

Thanks for the review [~xyao]!
{quote}FileContext assume to be a per process wide settings. If uri1 and uri2 are under different
context, should they use different Configuration objects? This way, the existing logic will
be able to handle it properly.
{quote}
That was my first approach too, but there are too many uses of FileContext (in YARN and potentially
as well as in downstream projects) and forcing all of them to create a new Configuration object
is not right. This creation of such a new Config object per URI was not needed in 2.x.

> After HADOOP-13440, multiple filesystems/file-contexts created with the same Configuration
object are forced to have the same umask
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-15571
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15571
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>            Priority: Critical
>         Attachments: HADOOP-15571.txt
>
>
> Ran into a super hard-to-debug issue due to this. [Edit: Turns out the same issue as
YARN-5749 that [~Tao Yang] ran into]
> h4. Issue
> Configuration conf = new Configuration();
>  fc1 = FileContext.getFileContext(uri1, conf);
>  fc2 = FileContext.getFileContext(uri2, conf);
>  fc.setUMask(umask_for_fc1); // Screws up umask for fc2 also!
> This was not the case before HADOOP-13440.
> h4. Symptoms:
> h5. Scenario I ran into
> When trying to localize a HDFS directory (hdfs:///my/dir/1.txt), NodeManager tries to
replicate the directory structure on the local file-system ($yarn-local-dirs/filecache/my/dir/1.txt).
> Now depending on whether NM has ever done a log-aggregation (completely unrelated code
that sets umask to be 137 for its own files on HDFS), the directories /my and /my/dir on local-fs
may have different permissions. In the specific case where NM did log-aggregation, /my/dir
was created with 137 umask and so localization of 1.txt completely failed due to absent directory
executable permissions!
> h5. Previous scenarios:
> We ran into this before in test-cases and instead of fixing the root-cause, we just fixed
the test-cases: YARN-5679 / YARN-5749



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message