hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Toshihiro Suzuki (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-17710) HBase in standalone mode creates directories with 777 permission
Date Wed, 01 Mar 2017 02:25:45 GMT
Toshihiro Suzuki created HBASE-17710:
----------------------------------------

             Summary: HBase in standalone mode creates directories with 777 permission
                 Key: HBASE-17710
                 URL: https://issues.apache.org/jira/browse/HBASE-17710
             Project: HBase
          Issue Type: Bug
          Components: regionserver
    Affects Versions: 1.1.2
         Environment: HDP-2.5.3
            Reporter: Toshihiro Suzuki


HBase in standalone mode creates directories with 777 permission in hbase.rootdir.
Ambari metrics collector defaults to standalone mode.

{code}
# find /var/lib/ambari-metrics-collector/hbase -perm 777 -type d -exec ls -ld {} \;
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/hbase/namespace/d0cca53847904f4b4add1caa0ce3a9af/info
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/hbase/backup/cbceb8fccd968b4b4583365d4dc6e377/meta
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/hbase/backup/cbceb8fccd968b4b4583365d4dc6e377/session
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.CATALOG/2f4ce2294cd21cecb58fd1aca5646144/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.SEQUENCE/0eb67274ece8a4a26cfeeef2c6d4cd37/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.SEQUENCE/aef86710a4005f98e2dc90675f2eb325/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.STATS/5b1d955e255e55979621214a7e4083b8/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.FUNCTION/32c033735cf144bac5637de23f7f7dd0/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRICS_METADATA/e420dfa799742fe4516ad1e4deefb793/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/HOSTED_APPS_METADATA/110be63e2a9994121fc5b48d663daf2c/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/CONTAINER_METRICS/a103719f87e8430635abf51a7fe98637/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD/cdb1d032beb90e350ce309e5d383c78e/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD/294deab47187494e845a5199702b4d04/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_MINUTE/1a263b4fe068ef2db5ba1c3e45553354/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_MINUTE/48f94dfb0161d8a28f645d2e1a473235/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_HOURLY/6d096ac3e70e54dd4a8612e17cfc4b11/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_DAILY/e81850d62da64c8d1c67be309f136e23/0
drwxrwxrwx. 2 ams hadoop 45 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/b43ff796de887197834ad62fdb612b59/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/b43ff796de887197834ad62fdb612b59/.tmp
drwxrwxrwx. 2 ams hadoop 45 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/c8eadeb7dead8fda9729b8e9b10c4929/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/c8eadeb7dead8fda9729b8e9b10c4929/.tmp
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_MINUTE/ca9f9754ae9ae4cdc3e1b0523eecc390/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:18 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_HOURLY/8412e8a8aec5d6307943fac78ce14c7a/0
drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:18 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_DAILY/7c3358aba91ea0d76ddd8bc3ceb2d578/0
{code}

My analysis is as follows:

FileSystem.mkdirs(Path f) method creates a directory with permission 777. Because HFileSystem
which inherits FileSystem doesn't override the method, when we call HFileSystem.mkdirs(Path
f), it tries to create a directory with permission 777.

I've found that HFileSystem.mkdirs(Path f) gets called in the following areas:
https://github.com/hortonworks/hbase/blob/HDP-2.5.3.0-37-tag/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L686
https://github.com/hortonworks/hbase/blob/HDP-2.5.3.0-37-tag/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java#L968

At these times, the call path is as follows ("fs" is an instance of the abstract FileSystem
class, being a field / member variable of FilterFileSystem):

HFileSystem.mkdirs(Path f) -> FileSystem.mkdirs(Path f) -> FilterFileSystem.mkdirs(Path
f, FsPermission permission) -> fs.mkdirs(Path f, FsPermission permission)

For example:
* In HRegionFileSystem.createDir(Path dir), fs.mkdirs(dir) is called. 
* Here "fs" is an instance of HFileSystem.
* HFileSystem who inherits from FileSystem does not have an overridden implementation of mkdirs(dir).
* Therefore, the parent FileSystem.mkdirs(Path f) is called.
* FileSystem.mkdirs(Path f) calls FileSystem.mkdirs(f, FsPermission.getDirDefault()).
** FsPermission.getDirDefault() returns a default permission of 777 via this line:
*** return new FsPermission((short)00777);
* FileSystem.mkdirs(f, FsPermission.getDirDefault()) is abstract, so FilterFileSystem.mkdirs(Path
f, FsPermission permission) gets called.
* FilterFileSystem.mkdirs(Path f, FsPermission permission) calls fs.mkdirs(f, permission).
* "fs" at this point can be either RawLocalFileSystem or DistributedFileSystem depending on
whether we are in standalone mode or distributed mode.

RawLocalFileSystem.mkdirs creates directories with permission 777. On the other hand, DistributedFileSystem.mkdirs
applies the umask to the 777 permission (https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Changes_to_the_File_System_API).
By default, the umask (fs.permissions.umask-mode) is 0022, so the permission of the directory
becomes 755 after application of the 0022 umask.

In distributed mode directories with 777 are not created in HDFS, but in standalone mode directories
with 777 are created.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message