hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17710) HBase in standalone mode creates directories with 777 permission
Date Thu, 02 Mar 2017 20:09:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15892891#comment-15892891
] 

Hudson commented on HBASE-17710:
--------------------------------

SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2600 (See [https://builds.apache.org/job/HBase-Trunk_matrix/2600/])
HBASE-17710 HBase in standalone mode creates directories with 777 (tedyu: rev 0b3ecc5ee7b990ef2149b388ed4f2eef4c5b139a)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java


> HBase in standalone mode creates directories with 777 permission
> ----------------------------------------------------------------
>
>                 Key: HBASE-17710
>                 URL: https://issues.apache.org/jira/browse/HBASE-17710
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 1.1.2
>         Environment: HDP-2.5.3
>            Reporter: Toshihiro Suzuki
>            Assignee: Ted Yu
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: 17710.branch-1.v1.txt, 17710.branch-1.v2.txt, 17710.branch-1.v2.txt,
17710.branch-1.v3.txt, 17710.v1.txt, 17710.v2.txt, 17710.v3.txt, 17710.v4.txt, 17710.v5.txt
>
>
> HBase in standalone mode creates directories with 777 permission in hbase.rootdir.
> Ambari metrics collector defaults to standalone mode.
> {code}
> # find /var/lib/ambari-metrics-collector/hbase -perm 777 -type d -exec ls -ld {} \;
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/hbase/namespace/d0cca53847904f4b4add1caa0ce3a9af/info
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/hbase/backup/cbceb8fccd968b4b4583365d4dc6e377/meta
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/hbase/backup/cbceb8fccd968b4b4583365d4dc6e377/session
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.CATALOG/2f4ce2294cd21cecb58fd1aca5646144/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.SEQUENCE/0eb67274ece8a4a26cfeeef2c6d4cd37/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.SEQUENCE/aef86710a4005f98e2dc90675f2eb325/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.STATS/5b1d955e255e55979621214a7e4083b8/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.FUNCTION/32c033735cf144bac5637de23f7f7dd0/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRICS_METADATA/e420dfa799742fe4516ad1e4deefb793/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/HOSTED_APPS_METADATA/110be63e2a9994121fc5b48d663daf2c/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/CONTAINER_METRICS/a103719f87e8430635abf51a7fe98637/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD/cdb1d032beb90e350ce309e5d383c78e/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD/294deab47187494e845a5199702b4d04/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_MINUTE/1a263b4fe068ef2db5ba1c3e45553354/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_MINUTE/48f94dfb0161d8a28f645d2e1a473235/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_HOURLY/6d096ac3e70e54dd4a8612e17cfc4b11/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_DAILY/e81850d62da64c8d1c67be309f136e23/0
> drwxrwxrwx. 2 ams hadoop 45 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/b43ff796de887197834ad62fdb612b59/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/b43ff796de887197834ad62fdb612b59/.tmp
> drwxrwxrwx. 2 ams hadoop 45 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/c8eadeb7dead8fda9729b8e9b10c4929/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/c8eadeb7dead8fda9729b8e9b10c4929/.tmp
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_MINUTE/ca9f9754ae9ae4cdc3e1b0523eecc390/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:18 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_HOURLY/8412e8a8aec5d6307943fac78ce14c7a/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:18 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_DAILY/7c3358aba91ea0d76ddd8bc3ceb2d578/0
> {code}
> My analysis is as follows:
> FileSystem.mkdirs(Path f) method creates a directory with permission 777. Because HFileSystem
which inherits FileSystem doesn't override the method, when we call HFileSystem.mkdirs(Path
f), it tries to create a directory with permission 777.
> I've found that HFileSystem.mkdirs(Path f) gets called in the following areas:
> https://github.com/hortonworks/hbase/blob/HDP-2.5.3.0-37-tag/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L686
> https://github.com/hortonworks/hbase/blob/HDP-2.5.3.0-37-tag/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java#L968
> At these times, the call path is as follows ("fs" is an instance of the abstract FileSystem
class, being a field / member variable of FilterFileSystem):
> HFileSystem.mkdirs(Path f) -> FileSystem.mkdirs(Path f) -> FilterFileSystem.mkdirs(Path
f, FsPermission permission) -> fs.mkdirs(Path f, FsPermission permission)
> For example:
> * In HRegionFileSystem.createDir(Path dir), fs.mkdirs(dir) is called. 
> * Here "fs" is an instance of HFileSystem.
> * HFileSystem who inherits from FileSystem does not have an overridden implementation
of mkdirs(dir).
> * Therefore, the parent FileSystem.mkdirs(Path f) is called.
> * FileSystem.mkdirs(Path f) calls FileSystem.mkdirs(f, FsPermission.getDirDefault()).
> ** FsPermission.getDirDefault() returns a default permission of 777 via this line:
> *** return new FsPermission((short)00777);
> * FileSystem.mkdirs(f, FsPermission.getDirDefault()) is abstract, so FilterFileSystem.mkdirs(Path
f, FsPermission permission) gets called.
> * FilterFileSystem.mkdirs(Path f, FsPermission permission) calls fs.mkdirs(f, permission).
> * "fs" at this point can be either RawLocalFileSystem or DistributedFileSystem depending
on whether we are in standalone mode or distributed mode.
> RawLocalFileSystem.mkdirs creates directories with permission 777. On the other hand,
DistributedFileSystem.mkdirs applies the umask to the 777 permission (https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Changes_to_the_File_System_API).
By default, the umask (fs.permissions.umask-mode) is 0022, so the permission of the directory
becomes 755 after application of the 0022 umask.
> In distributed mode directories with 777 are not created in HDFS, but in standalone mode
directories with 777 are created.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message