hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chunhui shen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7403) Online Merge
Date Sat, 02 Feb 2013 02:07:14 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569381#comment-13569381
] 

chunhui shen commented on HBASE-7403:
-------------------------------------

The above stack seems normal.

The failed test is 
{code}
  testMasterFailoverWithMockedRIT(org.apache.hadoop.hbase.master.TestMasterFailover): test
timed out after 180000 milliseconds
-1 core zombie tests{color}.  There are 1 zombie test(s):       at org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRIT(TestMasterFailover.java:391){code}
{code}

"Thread-613" prio=10 tid=0x766df800 nid=0xae3 in Object.wait() [0x770fe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on &lt;0x83a5cad8> (a org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
        at java.lang.Thread.join(Thread.java:1194)
        - locked &lt;0x83a5cad8> (a org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
        at org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:119)
        at org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:399)
        at org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:676)
        at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:895)
        at org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:863)
        at org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRIT(TestMasterFailover.java:391)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}

From{code}
Automatic Stack Trace every 60 seconds waiting on RegionServer:0;asf002.sp2.ygridcore.net,58219,1359710594587
{code}
We could see regionserver is hanged when stopping, causing test failed.

Through the Stack Trace every 60 seconds, we could find that RegionServer:0;asf002.sp2.ygridcore.net,58219,1359710594587
is hanged by the following stack:
{code}
NumberFormatException: Thread 1251 (RegionServer:0;asf002.sp2.ygridcore.net,58219,1359710594587):
  State: TIMED_WAITING
  Blocked count: 41
  Waited count: 1179
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:133)
    org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1127)
    org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:963)
    org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151)
    org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103)
    org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:135)
    java.security.AccessController.doPrivileged(Native Method)
    javax.security.auth.Subject.doAs(Subject.java:337)
    org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1118)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.apache.hadoop.hbase.util.Methods.call(Methods.java:41)
    org.apache.hadoop.hbase.security.User.call(User.java:404)
    org.apache.hadoop.hbase.security.User.access$300(User.java:52)
    org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:244)
    org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:133)
    java.lang.Thread.run(Thread.java:662)


NumberFormatException: Thread 1747 (RS_CLOSE_REGION-asf002.sp2.ygridcore.net,58219,1359710594587-0):
  State: RUNNABLE
  Blocked count: 5
  Waited count: 7
  Stack:
    java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
    java.util.TreeMap.deleteEntry(TreeMap.java:2151)
    java.util.TreeMap.remove(TreeMap.java:585)
    java.util.TreeSet.remove(TreeSet.java:259)
    org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:55)
    org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:86)
    org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:40)
    org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1072)
    org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:976)
    org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:146)
    org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:202)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
{code}

The above stack trace is kept same every 60 seconds, it seems Thread 1747 is always runnable
at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193). Are there too many entries in tree
map? 
It seems weird, let me try to reproduce this on my local PC



                
> Online Merge
> ------------
>
>                 Key: HBASE-7403
>                 URL: https://issues.apache.org/jira/browse/HBASE-7403
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 0.94.3
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>            Priority: Critical
>             Fix For: 0.96.0, 0.94.6
>
>         Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt,
7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch,
hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch,
merge region.pdf
>
>
> The feature of this online merge:
> 1.Online,no necessary to disable table
> 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
> 3.Easy to call merege request, no need to input a long region name, only encoded name
enough
> 4.No limit when operation, you don't need to tabke care the events like Server Dead,
Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge
request, it has alread done for you
> 5.Only little offline time for two merging regions
> Usage:
> 1.Tool:  
> bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] <table-name>
<region-encodedname-1> <region-encodedname-2>
> 2.API: static void MergeManager#createMergeRequest
> We need merge in the following cases:
> 1.Region hole or region overlap, can’t be fix by hbck
> 2.Region become empty because of TTL and not reasonable Rowkey design
> 3.Region is always empty or very small because of presplit when create table
> 4.Too many empty or small regions would reduce the system performance(e.g. mslab)
> Current merge tools only support offline and are not able to redo if exception is thrown
in the process of merging, causing a dirty data
> For online system, we need a online merge.
> This implement logic of this patch for  Online Merge is :
> For example, merge regionA and regionB into regionC
> 1.Offline the two regions A and B
> 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and
regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory)
> 3.Add the merged regionC to .META.
> 4.Assign the merged regionC
> As design of this patch , once we do the merge work in the HDFS,we could redo it until
successful if it throws exception or abort or server restart, but couldn’t be rolled back.

> It depends on
> Use zookeeper to record the transaction journal state, make redo easier
> Use zookeeper to send/receive merge request
> Merge transaction is executed on the master
> Support calling merge request through API or shell tool
> About the merge process, please see the attachment and patch

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message