hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Swarnim Kulkarni (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7116) HDFS FileSystem object cache causes permission issues in creating tmp directories
Date Thu, 29 May 2014 04:24:02 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012051#comment-14012051
] 

Swarnim Kulkarni commented on HIVE-7116:
----------------------------------------

Hey Vikram,

Looking deeper into the patch, in my opinion there could be some issues:

1. "hadoop-1" pulls in the 1.2.1 hadoop jar. The FileSystem object on this doesn't have the
newInstance method on it. So with "-P hadoop-1" this would fail with a compilation error.
2. I think the latest patch is overly complicated for what it is trying to do. I think it
could be simplified to just the following:

{code}
FileSystem fs = FileSystem.newInstance(mkdirPath.toUri(), conf);
    boolean retval = false;
    try {
      retval = fs.mkdirs(mkdirPath, fsPermission);
    } finally {
      org.apache.commons.io.IOUtils.closeQuietly(fs);
    }
{code}

3. Are we sure that it is the cached FileSystem instance that is causing the permissions to
not get reflected? I wrote a simple test with cached FileSystem object[1] that sets permissions
and it passed for me. Interestingly it only fails for "777" permissions so might be some issue
specific to that additional "executable" sticky bit.

Just my 0.02 :)

[1] http://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/fs/FileSystem.html
[2] https://gist.github.com/swarnim87/afd29a06e6cc7fffae5c

> HDFS FileSystem object cache causes permission issues in creating tmp directories
> ---------------------------------------------------------------------------------
>
>                 Key: HIVE-7116
>                 URL: https://issues.apache.org/jira/browse/HIVE-7116
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2, Tez
>    Affects Versions: 0.13.0
>            Reporter: Vikram Dixit K
>            Assignee: Vikram Dixit K
>         Attachments: HIVE-7116.1.patch, HIVE-7116.2.patch, HIVE-7116.3.patch, HIVE-7116.4.patch
>
>
> We change permissions of the directory creation to 777 for HiveServer 2 operation and
it turns out that because of HDFS caching, it does not reflect once created. We need to use
the non-cached version of the API to get a FileSystem object to fix this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message