hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1020) Resource Localization using Groups as a new Localization Type
Date Fri, 02 Aug 2013 21:01:49 GMT

    [ https://issues.apache.org/jira/browse/YARN-1020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13728110#comment-13728110
] 

Jason Lowe commented on YARN-1020:
----------------------------------

A container could modify a localized resource if the permissions of the localized resource
file allow it.  For example, one could place an archive in the distributed cache and that
archive could contain a file with permissions like rw-rw-rw-.  With those permissions, any
container could change the file and any other container would see those changes until the
resource is flushed from the cache of that node.  Similarly if the archive contained a directory
with sufficient write permissions, other containers using that resource could reach in and
deposit new files or delete existing files.

One possible approach is to explicitly scrub the ownership and permissions of files/directories
for a PUBLIC resource when it is localized to remove all write permissions.  Allowing a localized
resource to be modified, even by the original application that requested it, can cause all
sorts of sporadic bugs.  The remove-write-permissions scrub would enforce the assumption that
the resource is read-only, and since PUBLIC files are localized as the NM user and not the
app user they shouldn't be able to change the permissions back once localized.
                
> Resource Localization using Groups as a new Localization Type
> -------------------------------------------------------------
>
>                 Key: YARN-1020
>                 URL: https://issues.apache.org/jira/browse/YARN-1020
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Omkar Vinit Joshi
>
> The scenario is as follows..
> * We definitely will have multiple applications running on top of yarn. These applications
whenever run by users will need resources to be localized. Now the options what application-users
will have for localizing resources are:-
> ** APPLICATION ... these files will be available for only that instance of the application
and only for that single user. If we talk in terms of MR then for single job.
> ** PRIVATE ... available only for that user only for multiple runs of that application.
Other users clearly will not be able to take advantage of that. So ideally will be wasting
space (local resource cache) by replicating the same file again and again.
> ** PUBLIC... there will be only one copy of individual files of the application say APP_1..GOOD
..in the sense it will be accessible to all the users...But for secured clusters; users of
different application (say APP_2) containers can then gain easy access to this applications
(APP_1) private files and potentially may modify it.
> So clearly we don't have any solution today to solve the above problem with existing
RESOURCE_LOCALIZATION_TYPES without effectively using space. Therefore we need something like
GROUP to address this scenario.
> Thoughts?? 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message