hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhihong Yu (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
Date Thu, 19 Jan 2012 18:46:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189277#comment-13189277
] 

Zhihong Yu commented on HBASE-5179:
-----------------------------------

{code}
2012-01-19 11:10:35,847 INFO org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
ephemeral node deleted, processing expiration [C3S31,20020,1326987900492]
2012-01-19 11:10:35,850 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=C3S31,20020,1326987900492
to dead servers, submitted shutdown handler to be executed, root=true, meta=false
2012-01-19 11:10:35,851 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Splitting logs for C3S31,20020,1326987900492
...
2012-01-19 11:12:37,003 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://C3S31:9000/hbase/.logs/C3S31,20020,1326987900492/C3S31%3A20020.1326987901100
to hdfs://C3S31:9000/hbase/.oldlogs/C3S31%3A20020.1326987901100
2012-01-19 11:12:37,004 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file
splitting completed in 1137 ms for hdfs://C3S31:9000/hbase/.logs/C3S31,20020,1326987900492
2012-01-19 11:12:37,004 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Server C3S31,20020,1326987900492 was carrying ROOT. Trying to assign.
...
2012-01-19 11:12:37,382 INFO org.apache.hadoop.hbase.master.HMaster: .META. assigned=2, rit=false,
location=C3S32:20020
2012-01-19 11:12:37,382 INFO org.apache.hadoop.hbase.master.HMaster: Master startup proceeding:
master failover
2012-01-19 11:12:37,383 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
1028785192 on serverName=C3S32,20020,1326987400439, load=(requests=0, regions=0, usedHeap=0,
maxHeap=0)
2012-01-19 11:12:37,384 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
70236052 on serverName=C3S32,20020,1326987400439, load=(requests=0, regions=0, usedHeap=0,
maxHeap=0)
2012-01-19 11:12:37,393 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Reassigning 0 region(s) that C3S31,20020,1326987900492 was carrying (skipping 0 regions(s)
that are already in transition)
2012-01-19 11:12:37,393 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Finished processing of shutdown of C3S31,20020,1326987900492
{code}
>From above log, we can see that .META. was assigned to C3S32 and log splitting was for
C3S31.

So was there data loss ?

                
> Concurrent processing of processFaileOver and ServerShutdownHandler may cause region
to be assigned before log splitting is completed, causing data loss
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-5179
>                 URL: https://issues.apache.org/jira/browse/HBASE-5179
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.90.2
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>            Priority: Critical
>             Fix For: 0.92.0, 0.94.0, 0.90.6
>
>         Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 5179-90v12.patch,
5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch,
5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt,
5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, hbase-5179v5.patch,
hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch
>
>
> If master's processing its failover and ServerShutdownHandler's processing happen concurrently,
it may appear following  case.
> 1.master completed splitLogAfterStartup()
> 2.RegionserverA restarts, and ServerShutdownHandler is processing.
> 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server.
> 4.master starts to assign regions of RegionserverA because it is a dead server by step3.
> However, when doing step4(assigning region), ServerShutdownHandler may be doing split
log, Therefore, it may cause data loss.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message