hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6160) META entries from daughters can be deleted before parent entries
Date Mon, 04 Jun 2012 21:57:23 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288941#comment-13288941
] 

Enis Soztutar commented on HBASE-6160:
--------------------------------------

The exception: 
{code}
12/06/04 06:50:41 ERROR security.UserGroupInformation: PriviledgedActionException as: cause:org.apache.hadoop.hbase.client.RegionOfflineException:
Split daughter region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
cannot be found in META.
Exception in thread "main" org.apache.hadoop.hbase.client.RegionOfflineException: Split daughter
region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
cannot be found in META.
        at org.apache.hadoop.hbase.client.MetaScanner$BlockingMetaScannerVisitor.processRow(MetaScanner.java:433)
        at org.apache.hadoop.hbase.client.MetaScanner$TableMetaScannerVisitor.processRow(MetaScanner.java:490)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:227)
        at org.apache.hadoop.hbase.client.MetaScanner.access$000(MetaScanner.java:57)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:136)
        at org.apache.hadoop.hbase.client.MetaScanner$1.connect(MetaScanner.java:133)
        at org.apache.hadoop.hbase.client.HConnectionManager.execute(HConnectionManager.java:361)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:133)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:108)
        at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:86)
        at org.apache.hadoop.hbase.client.MetaScanner.allTableRegions(MetaScanner.java:326)
        at org.apache.hadoop.hbase.client.HTable.getRegionLocations(HTable.java:499)
        at org.apache.hadoop.hbase.client.HTable.getStartEndKeys(HTable.java:452)
        at org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:132)
        at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:962)
        at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:979)

{code}

So the region in question is 
{code}
50a4617eead34cad335a8dfa727d177d
{code}
and from the logs we see that {{25d9c4ff574a37bd95bf5e5be6d618dd}} is split into {{1dc74065583c67b3916c4ed158cb53fa}}
and {{50a4617eead34cad335a8dfa727d177d}}

{code}
./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 04:56:02,855 INFO org.apache.hadoop.hbase.regionserver.SplitRequest:
Region split, META updated, and report to master. Parent=TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.,
new regions: TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa.,
TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d..
Split took 4sec
{code}

After some time, {{50a4617eead34cad335a8dfa727d177d}} is further split into two: 

{code}
./hbase-hbase-regionserver-ip-10-226-65-102.log:2012-06-04 05:41:13,488 INFO org.apache.hadoop.hbase.regionserver.SplitRequest:
Region split, META updated, and report to master. Parent=TestLoadAndVerify_1
338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.,
new regions: TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338802866393.16288
65d7fa8e9eec3a7d8073465296e., TestLoadAndVerify_1338798130970,]y\x04\x00\x00\x00\x00\x00/000047_0,1338802866393.413cafe6c61426e26254c197e8c0a6ba..
Split took 7sec
{code}

Further time passes, and CatalogJanitor deletes the META entry for that region:
{code}
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:16,688 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Deleting region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
because daughter splits no longer hold references
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Deleted daughters references, qualifier=splitA and qualifier=splitB, from parent TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,103 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
DELETING region hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:47:18,145 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Deleted region TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.
from META
./hbase-hbase-master-ip-10-144-69-91.log:2012-06-04 05:52:12,891 WARN org.apache.hadoop.hbase.master.CatalogJanitor:
Daughter regiondir does not exist: hdfs://ip-10-10-50-98.ec2.internal:8020/apps/hbase/data/TestLoadAndVerify_1338798130970/50a4617eead34cad335a8dfa727d177d
{code}

However, there is no log for the grantfather,25d9c4ff574a37bd95bf5e5be6d618dd, being deleted
from META. In fact, META scan also confirms that the grandfather is in META, but not the father:

{code}
 TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:regioninfo, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.',
STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl',
ENCODED => 25d9c4ff574a37bd95bf5e5be6d618dd, OFFLINE => true, SPLIT => true,}
 TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:server, timestamp=1338799025521, value=ip-10-226-65-102.ec2.internal:60020
 TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:serverstartcode, timestamp=1338799025521, value=1338780117340
 TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:splitA, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338800158687.1dc74065583c67b3916c4ed158cb53fa.',
STARTKEY => '[\x02\x01\x00\x00\x00\x00\x00/000071_0', ENDKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0',
ENCODED => 1dc74065583c67b3916c4ed158cb53fa,}
 TestLoadAndVerify_1338798130970,[\x02\x01\x00\x00\x00\x00\x00/000071_0,1338799021182.25d9c4ff574a37bd95bf5e5be6d618dd.
column=info:splitB, timestamp=1338800161105, value={NAME => 'TestLoadAndVerify_1338798130970,\\xA2\x04\x00\x00\x00\x00\x00/000048_0,1338800158687.50a4617eead34cad335a8dfa727d177d.',
STARTKEY => '\\xA2\x04\x00\x00\x00\x00\x00/000048_0', ENDKEY => '^P\xD7\x945\xE5\x0Dl',
ENCODED => 50a4617eead34cad335a8dfa727d177d,}
{code}

                
> META entries from daughters can be deleted before parent entries
> ----------------------------------------------------------------
>
>                 Key: HBASE-6160
>                 URL: https://issues.apache.org/jira/browse/HBASE-6160
>             Project: HBase
>          Issue Type: Bug
>          Components: client, regionserver
>    Affects Versions: 0.92.2, 0.94.0, 0.96.0
>            Reporter: Enis Soztutar
>            Assignee: Enis Soztutar
>
> HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but
not the children. However, after the fix, we have seen the following issue in tests: 
> Region A is split to -> B, C
> Region B is split to -> D, E
> After some time, META entry for B is deleted since it is not needed anymore, but META
entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException
for B. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message