hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Clay B. (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-12465) HBase master start fails due to incorrect file creations
Date Mon, 08 Jun 2015 04:11:01 GMT

    [ https://issues.apache.org/jira/browse/HBASE-12465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576581#comment-14576581
] 

Clay B. commented on HBASE-12465:
---------------------------------

If of use, Jeffrey, Alicia, et al. the errors we (Biju and Sudarshan) saw were:

Fatal to the running HMaster as well as any HMaster starting:
{code}
2015-06-06 02:08:52,374 ERROR org.apache.hadoop.hbase.backup.HFileArchiver: Failed to archive
class org.apache.hadoop.hbase.backup.HFi
leArchiver$FileablePath, file:hdfs://cluster1/hbase/.tmp/data/default/xyxyTablenamE12345/002cd7fbf10def3bb3149ed85707fabf/e/f2806ebb34ab493ebe4b623fac585776_SeqId_385900210_
2015-06-06 02:08:52,374 WARN org.apache.hadoop.hbase.backup.HFileArchiver: Couldn't archive
class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://cluster1/hbase/.tmp/data/default/xyxyTablenamE12345/002cd7fbf10def3bb3149ed85707fabf/e/f2806ebb34ab493ebe4b623fac585776_SeqId_385900210_
into backup directory: hdfs://cluster1/hbase/archive/data/default/xyxyTablenamE12345/002cd7fbf10def3bb3149ed85707fabf/e
2015-06-06 02:08:52,374 WARN org.apache.hadoop.hbase.backup.HFileArchiver: Failed to complete
archive of: [class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://cluster1/hbase/.tmp/data/default/xyxyTablenamE12345/002cd7fbf10def3bb3149ed85707fabf/e/f2806ebb34ab493ebe4b623fac585776_SeqId_385900210_].
Those files are still in the original location, and they may slow down reads.
2015-06-06 02:08:52,374 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception.
Starting shutdown.
java.io.IOException: Received error when attempting to archive files ([class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath,
file:hdfs://cluster1/hbase/.tmp/data/default/xyxyTablenamE12345/002cd7fbf10def3bb3149ed85707fabf/e]),
cannot delete region directory.
        at org.apache.hadoop.hbase.backup.HFileArchiver.archiveRegion(HFileArchiver.java:148)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkTempDir(MasterFileSystem.java:503)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:149)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:127)
        at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:789) 

        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:606)
        at java.lang.Thread.run(Thread.java:745)  
2015-06-06 02:08:52,375 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2015-06-06 02:08:52,375 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
[... more shutting down here ...]
2015-06-06 02:28:15,583 DEBUG org.apache.hadoop.hbase.master.ActiveMasterManager: A master
is now available
2015-06-06 02:28:15,583 INFO org.apache.hadoop.hbase.master.ActiveMasterManager: Registered
Active Master=cluster1-bcpc-r2n7.example.com,60000,1433572093679
2015-06-06 02:28:15,588 INFO org.apache.hadoop.conf.Configuration.deprecation: fs.default.name
is deprecated. Instead, use fs.defaultFS
2015-06-06 02:28:15,968 INFO org.apache.hadoop.conf.Configuration.deprecation: hadoop.native.lib
is deprecated. Instead, use io.native.lib.available
2015-06-06 02:28:16,164 DEBUG org.apache.hadoop.hbase.util.FSTableDescriptors: Current tableInfoPath
= hdfs://cluster1/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
2015-06-06 02:28:16,195 DEBUG org.apache.hadoop.hbase.util.FSTableDescriptors: TableInfo already
exists.. Skipping creation
2015-06-06 02:28:17,850 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: ARCHIVING hdfs://cluster1/hbase/.tmp/data/default/xyxyTablenamE12345/002cd7fbf10def3bb3149ed85707fabf
2015-06-06 02:28:17,860 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Archiving [class
org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://cluster1/hbase/.tmp/data/default/xyxyTablenamE12345/002cd7fbf10def3bb3149ed85707fabf/e]
2015-06-06 02:28:17,885 WARN org.apache.hadoop.hbase.backup.HFileArchiver: Failed to archive
class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://cluster1/hbase/.tmp/data/default/xyxyTablenamE12345/002cd7fbf10def3bb3149ed85707fabf/e/f2806ebb34ab493ebe4b623fac585776_SeqId_385900210_
on try #0
org.apache.hadoop.security.AccessControlException: Permission denied: user=hbase, access=WRITE,
inode="/hbase/.tmp/data/default/xyxyTablenamE12345/002cd7fbf10def3bb3149ed85707fabf/e/f2806ebb34ab493ebe4b623fac585776_SeqId_385900210_":userName:supergroup:-rwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:234)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:164)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5202)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5184)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:5146)
[... more file permission failures ...]
{code}

> HBase master start fails due to incorrect file creations
> --------------------------------------------------------
>
>                 Key: HBASE-12465
>                 URL: https://issues.apache.org/jira/browse/HBASE-12465
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.96.0
>         Environment: Ubuntu
>            Reporter: Biju Nair
>            Assignee: Alicia Ying Shu
>              Labels: hbase, hbase-bulkload
>
> - Start of HBase master fails due to the following error found in the log.
> 2014-11-11 20:25:58,860 WARN org.apache.hadoop.hbase.backup.HFileArchiver: 
> Failed to archive class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePa
> th,file:hdfs://YYYY/hbase/.tmp/data/default/tbl/00820520f5cb7839395e83f40c8d97c2/e/52bf9eee7a27460c8d9e2a26fa43c918_SeqId_282271246_
on try #1
> org.apache.hadoop.security.AccessControlException: Permission denied: 
> user=hbase,access=WRITE,inode="/hbase/.tmp/data/default/tbl/00820520f5cb7839395e83f40c8d97c2/e/52bf9eee7a27460c8d9e2a26fa43c918_SeqId_282271246_":devuser:supergroup:-rwxr-xr-x
> -  All the files that hbase master was complaining about are created under an users user-id
instead on "hbase" user resulting in incorrect access permission for the master to act on.
> - Looks like this was due to bulk load done using LoadIncrementalHFiles program.
> - HBASE-12052 is another scenario similar to this one. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message