hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-17710) HBase in standalone mode creates directories with 777 permission
Date Thu, 02 Mar 2017 12:28:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-17710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15892147#comment-15892147
] 

Hadoop QA commented on HBASE-17710:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue}
Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 4s {color} | {color:blue}
The patch file was not named according to hbase's naming conventions. Please see https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} |
{color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red}
The patch doesn't appear to include any new or modified tests. Please justify why no new tests
are needed for this patch. Also please list what manual steps were performed to verify this
patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 23s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 43s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s {color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 27m 15s {color}
| {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5
2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 49s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 90m 27s {color} | {color:green}
hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s {color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 9s {color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12855596/17710.v5.txt
|
| JIRA Issue | HBASE-17710 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  hbaseanti  checkstyle
 compile  |
| uname | Linux 6e939b592700 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 697a55a |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/5919/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/5919/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> HBase in standalone mode creates directories with 777 permission
> ----------------------------------------------------------------
>
>                 Key: HBASE-17710
>                 URL: https://issues.apache.org/jira/browse/HBASE-17710
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 1.1.2
>         Environment: HDP-2.5.3
>            Reporter: Toshihiro Suzuki
>            Assignee: Ted Yu
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: 17710.branch-1.v1.txt, 17710.branch-1.v2.txt, 17710.branch-1.v2.txt,
17710.v1.txt, 17710.v2.txt, 17710.v3.txt, 17710.v4.txt, 17710.v5.txt
>
>
> HBase in standalone mode creates directories with 777 permission in hbase.rootdir.
> Ambari metrics collector defaults to standalone mode.
> {code}
> # find /var/lib/ambari-metrics-collector/hbase -perm 777 -type d -exec ls -ld {} \;
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/hbase/namespace/d0cca53847904f4b4add1caa0ce3a9af/info
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/hbase/backup/cbceb8fccd968b4b4583365d4dc6e377/meta
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/hbase/backup/cbceb8fccd968b4b4583365d4dc6e377/session
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.CATALOG/2f4ce2294cd21cecb58fd1aca5646144/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.SEQUENCE/0eb67274ece8a4a26cfeeef2c6d4cd37/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.SEQUENCE/aef86710a4005f98e2dc90675f2eb325/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.STATS/5b1d955e255e55979621214a7e4083b8/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/SYSTEM.FUNCTION/32c033735cf144bac5637de23f7f7dd0/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRICS_METADATA/e420dfa799742fe4516ad1e4deefb793/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/HOSTED_APPS_METADATA/110be63e2a9994121fc5b48d663daf2c/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/CONTAINER_METRICS/a103719f87e8430635abf51a7fe98637/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD/cdb1d032beb90e350ce309e5d383c78e/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD/294deab47187494e845a5199702b4d04/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_MINUTE/1a263b4fe068ef2db5ba1c3e45553354/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_MINUTE/48f94dfb0161d8a28f645d2e1a473235/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_HOURLY/6d096ac3e70e54dd4a8612e17cfc4b11/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD_DAILY/e81850d62da64c8d1c67be309f136e23/0
> drwxrwxrwx. 2 ams hadoop 45 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/b43ff796de887197834ad62fdb612b59/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/b43ff796de887197834ad62fdb612b59/.tmp
> drwxrwxrwx. 2 ams hadoop 45 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/c8eadeb7dead8fda9729b8e9b10c4929/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:21 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE/c8eadeb7dead8fda9729b8e9b10c4929/.tmp
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:17 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_MINUTE/ca9f9754ae9ae4cdc3e1b0523eecc390/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:18 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_HOURLY/8412e8a8aec5d6307943fac78ce14c7a/0
> drwxrwxrwx. 2 ams hadoop 6 Mar  1 02:18 /var/lib/ambari-metrics-collector/hbase/data/default/METRIC_AGGREGATE_DAILY/7c3358aba91ea0d76ddd8bc3ceb2d578/0
> {code}
> My analysis is as follows:
> FileSystem.mkdirs(Path f) method creates a directory with permission 777. Because HFileSystem
which inherits FileSystem doesn't override the method, when we call HFileSystem.mkdirs(Path
f), it tries to create a directory with permission 777.
> I've found that HFileSystem.mkdirs(Path f) gets called in the following areas:
> https://github.com/hortonworks/hbase/blob/HDP-2.5.3.0-37-tag/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L686
> https://github.com/hortonworks/hbase/blob/HDP-2.5.3.0-37-tag/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java#L968
> At these times, the call path is as follows ("fs" is an instance of the abstract FileSystem
class, being a field / member variable of FilterFileSystem):
> HFileSystem.mkdirs(Path f) -> FileSystem.mkdirs(Path f) -> FilterFileSystem.mkdirs(Path
f, FsPermission permission) -> fs.mkdirs(Path f, FsPermission permission)
> For example:
> * In HRegionFileSystem.createDir(Path dir), fs.mkdirs(dir) is called. 
> * Here "fs" is an instance of HFileSystem.
> * HFileSystem who inherits from FileSystem does not have an overridden implementation
of mkdirs(dir).
> * Therefore, the parent FileSystem.mkdirs(Path f) is called.
> * FileSystem.mkdirs(Path f) calls FileSystem.mkdirs(f, FsPermission.getDirDefault()).
> ** FsPermission.getDirDefault() returns a default permission of 777 via this line:
> *** return new FsPermission((short)00777);
> * FileSystem.mkdirs(f, FsPermission.getDirDefault()) is abstract, so FilterFileSystem.mkdirs(Path
f, FsPermission permission) gets called.
> * FilterFileSystem.mkdirs(Path f, FsPermission permission) calls fs.mkdirs(f, permission).
> * "fs" at this point can be either RawLocalFileSystem or DistributedFileSystem depending
on whether we are in standalone mode or distributed mode.
> RawLocalFileSystem.mkdirs creates directories with permission 777. On the other hand,
DistributedFileSystem.mkdirs applies the umask to the 777 permission (https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html#Changes_to_the_File_System_API).
By default, the umask (fs.permissions.umask-mode) is 0022, so the permission of the directory
becomes 755 after application of the 0022 umask.
> In distributed mode directories with 777 are not created in HDFS, but in standalone mode
directories with 777 are created.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message