hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6811) [ATS1.5] All history logs should be kept under its own User Directory.
Date Tue, 01 Aug 2017 22:32:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-6811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109903#comment-16109903
] 

Junping Du commented on YARN-6811:
----------------------------------

Thanks [~rohithsharma] for contributing the patch! The approach here looks general good to
me. The only concern here is it could be a bit performance impact as it will search two directories
(with User and without user). I think one improve could we don't search user directory when
"keep-under-user-dir" set to false. The verse is not true because we need to handle rolling
upgrade case.

Some detail comments:

{noformat}
public static final String
+      TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_KEEP_UNDER_USER_DIR =
+      TIMELINE_SERVICE_ENTITYGROUP_FS_STORE_PREFIX + "keep-under-user-dir"
{noformat}
The name of new added configuration is too long, can it simply be "with-user-dir"?

We should document the new configuration in yarn-default.xml with proper explanation of how
this configuration is used for.

Like my comments offline, {{createUserDir(String user)}} should have a better name given it
doesn't already create user dir (depends on configuration). May be better to call it {{getAppRootDir()}}?

We need to handle rolling upgrade case. I think we can add a unit test here as we can write
app log with "keep-under-user-dir" = false for writing to old location, and try to read it
out when set "keep-under-user-dir" = true.

> [ATS1.5]  All history logs should be kept under its own User Directory.
> -----------------------------------------------------------------------
>
>                 Key: YARN-6811
>                 URL: https://issues.apache.org/jira/browse/YARN-6811
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: timelineclient, timelineserver
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>         Attachments: YARN-6811.01.patch
>
>
> ATS1.5 allows to store history data in underlying FileSystem folder path i.e */acitve-dir*
and */done-dir*. These base directories are protected for unauthorized user access for other
users data by setting sticky bit for /active-dir. 
> But object store filesystems such as WASB does not have user access control on folders
and files. When WASB are used as underlying file system for ATS1.5, the history data which
are stored in FS are accessible to all users. *This would be a security risk*
> I would propose to keep history data under its own user directory i.e */active-dir/$USER*.
Even this do not solve basic user access from FS, but it provides capability to plugin Apache
Ranger policies for each user folders. One thing to note that setting policies to each user
folder is admin responsibility. But grouping all history data of one user folder allows to
set policies so that user access control is achieved. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message