hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7116) HDFS FileSystem object cache causes permission issues in creating tmp directories
Date Thu, 29 May 2014 06:30:03 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012117#comment-14012117
] 

Thejas M Nair commented on HIVE-7116:
-------------------------------------

[~swarnim] Good catch about the hadoop-1 compilation issue. [~vikram.dixit] To work around
that you can set ""fs.%s.impl.disable.cache"=true in conf, and then call FileSystem.get(URI
uri, Configuration conf)

[~swarnim] Regarding 2,  I think it is important to re-set the conf to its old state in case
of a failure as well.

[~vikram.dixit] You can use a function to avoid code duplication within the finally block
and outside. I think this log message is better logged at DEBUG level as it is also being
called for every vertex creation in Tez mode.

[~swarnim] Regarding 3, try your example with a path name where more than one level in dir
structure needs to be created "eg "/tmp/a/b/c". You will notice that only c gets created with
FsPermission value. That is why the umask needs to be set in config so that "a" and "b" also
get created with appropriate permissions. The cache becomes an issue only if you already have
a FileSystem object in cache with old config. Your example is not resulting in a FileSystem
object in cache with old config.







> HDFS FileSystem object cache causes permission issues in creating tmp directories
> ---------------------------------------------------------------------------------
>
>                 Key: HIVE-7116
>                 URL: https://issues.apache.org/jira/browse/HIVE-7116
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2, Tez
>    Affects Versions: 0.13.0
>            Reporter: Vikram Dixit K
>            Assignee: Vikram Dixit K
>         Attachments: HIVE-7116.1.patch, HIVE-7116.2.patch, HIVE-7116.3.patch, HIVE-7116.4.patch
>
>
> We change permissions of the directory creation to 777 for HiveServer 2 operation and
it turns out that because of HDFS caching, it does not reflect once created. We need to use
the non-cached version of the API to get a FileSystem object to fix this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message