hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mithun Radhakrishnan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-17791) Temp dirs under the staging directory should honour `inheritPerms`
Date Thu, 12 Oct 2017 21:01:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-17791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mithun Radhakrishnan updated HIVE-17791:
----------------------------------------
    Description: 
For [~cdrome]:

CLI creates two levels of staging directories but calls setPermissions on the top-level directory
only if {{hive.warehouse.subdir.inherit.perms=true}}.

The top-level directory, {{/user/cdrome/hive/words_text_dist/dt=c/.hive-staging_hive_2016-07-15_08-44-22_082_5534649671389063929-1}}
is created the first time {{Context.getExternalTmpPath}} is called.

The child directory, {{/user/cdrome/hive/words_text_dist/dt=c/.hive-staging_hive_2016-07-15_08-44-22_082_5534649671389063929-1/_tmp.-ext-10000}}
is created when {{TezTask.execute}} is called at line 164:

{code:java}
DAG dag = build(jobConf, work, scratchDir, appJarLr, additionalLr, ctx);
{code}

This calls {{DagUtils.createVertex}}, which calls {{Utilities.createTmpDirs}}:

{code:java}
3770   private static void createTmpDirs(Configuration conf,
3771       List<Operator<? extends OperatorDesc>> ops) throws IOException {
3772 
3773     while (!ops.isEmpty()) {
3774       Operator<? extends OperatorDesc> op = ops.remove(0);
3775 
3776       if (op instanceof FileSinkOperator) {
3777         FileSinkDesc fdesc = ((FileSinkOperator) op).getConf();
3778         Path tempDir = fdesc.getDirName();
3779 
3780         if (tempDir != null) {
3781           Path tempPath = Utilities.toTempPath(tempDir);
3782           FileSystem fs = tempPath.getFileSystem(conf);
3783           fs.mkdirs(tempPath); // <------ HERE!
3784         }
3785       }
3786 
3787       if (op.getChildOperators() != null) {
3788         ops.addAll(op.getChildOperators());
3789       }
3790     }
3791   }
{code}

It turns out that {{inheritPerms}} is no longer part of {{master}}. I'll rebase this for {{branch-2}},
and {{branch-2.2}}. {{master}} will have to wait till the issues around {{StorageBasedAuthProvider}},
directory permissions, etc. are sorted out.

(Note to self: YHIVE-857)

  was:
For [~cdrome]:

CLI creates two levels of staging directories but calls setPermissions on the top-level directory
only if {{hive.warehouse.subdir.inherit.perms=true}}.

The top-level directory, {{/user/cdrome/hive/words_text_dist/dt=c/.hive-staging_hive_2016-07-15_08-44-22_082_5534649671389063929-1}}
is created the first time {{Context.getExternalTmpPath}} is called.

The child directory, {{/user/cdrome/hive/words_text_dist/dt=c/.hive-staging_hive_2016-07-15_08-44-22_082_5534649671389063929-1/_tmp.-ext-10000}}
is created when {{TezTask.execute}} is called at line 164:

{code:java}
DAG dag = build(jobConf, work, scratchDir, appJarLr, additionalLr, ctx);
{code}

This calls {{DagUtils.createVertex}}, which calls {{Utilities.createTmpDirs}}:

{code:java}
3770   private static void createTmpDirs(Configuration conf,
3771       List<Operator<? extends OperatorDesc>> ops) throws IOException {
3772 
3773     while (!ops.isEmpty()) {
3774       Operator<? extends OperatorDesc> op = ops.remove(0);
3775 
3776       if (op instanceof FileSinkOperator) {
3777         FileSinkDesc fdesc = ((FileSinkOperator) op).getConf();
3778         Path tempDir = fdesc.getDirName();
3779 
3780         if (tempDir != null) {
3781           Path tempPath = Utilities.toTempPath(tempDir);
3782           FileSystem fs = tempPath.getFileSystem(conf);
3783           fs.mkdirs(tempPath); // <------ HERE!
3784         }
3785       }
3786 
3787       if (op.getChildOperators() != null) {
3788         ops.addAll(op.getChildOperators());
3789       }
3790     }
3791   }
{code}

It turns out that {{inheritPerms}} is no longer part of {{master}}. I'll rebase this for {{branch-2}},
and {{branch-2.2}}. {{master}} will have to wait till the issues around {{StorageBasedAuthProvider}},
directory permissions, etc. are sorted out.


> Temp dirs under the staging directory should honour `inheritPerms`
> ------------------------------------------------------------------
>
>                 Key: HIVE-17791
>                 URL: https://issues.apache.org/jira/browse/HIVE-17791
>             Project: Hive
>          Issue Type: Bug
>          Components: Authorization
>    Affects Versions: 2.2.0, 2.4.0
>            Reporter: Mithun Radhakrishnan
>            Assignee: Chris Drome
>         Attachments: HIVE-17791.1-branch-2.2.patch, HIVE-17791.1-branch-2.patch
>
>
> For [~cdrome]:
> CLI creates two levels of staging directories but calls setPermissions on the top-level
directory only if {{hive.warehouse.subdir.inherit.perms=true}}.
> The top-level directory, {{/user/cdrome/hive/words_text_dist/dt=c/.hive-staging_hive_2016-07-15_08-44-22_082_5534649671389063929-1}}
is created the first time {{Context.getExternalTmpPath}} is called.
> The child directory, {{/user/cdrome/hive/words_text_dist/dt=c/.hive-staging_hive_2016-07-15_08-44-22_082_5534649671389063929-1/_tmp.-ext-10000}}
is created when {{TezTask.execute}} is called at line 164:
> {code:java}
> DAG dag = build(jobConf, work, scratchDir, appJarLr, additionalLr, ctx);
> {code}
> This calls {{DagUtils.createVertex}}, which calls {{Utilities.createTmpDirs}}:
> {code:java}
> 3770   private static void createTmpDirs(Configuration conf,
> 3771       List<Operator<? extends OperatorDesc>> ops) throws IOException
{
> 3772 
> 3773     while (!ops.isEmpty()) {
> 3774       Operator<? extends OperatorDesc> op = ops.remove(0);
> 3775 
> 3776       if (op instanceof FileSinkOperator) {
> 3777         FileSinkDesc fdesc = ((FileSinkOperator) op).getConf();
> 3778         Path tempDir = fdesc.getDirName();
> 3779 
> 3780         if (tempDir != null) {
> 3781           Path tempPath = Utilities.toTempPath(tempDir);
> 3782           FileSystem fs = tempPath.getFileSystem(conf);
> 3783           fs.mkdirs(tempPath); // <------ HERE!
> 3784         }
> 3785       }
> 3786 
> 3787       if (op.getChildOperators() != null) {
> 3788         ops.addAll(op.getChildOperators());
> 3789       }
> 3790     }
> 3791   }
> {code}
> It turns out that {{inheritPerms}} is no longer part of {{master}}. I'll rebase this
for {{branch-2}}, and {{branch-2.2}}. {{master}} will have to wait till the issues around
{{StorageBasedAuthProvider}}, directory permissions, etc. are sorted out.
> (Note to self: YHIVE-857)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message