hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhihong Yu (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5179) Concurrent processing of processFaileOver and ServerShutdownHandler may cause region to be assigned before log splitting is completed, causing data loss
Date Thu, 19 Jan 2012 18:46:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189277#comment-13189277

Zhihong Yu commented on HBASE-5179:

2012-01-19 11:10:35,847 INFO org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer
ephemeral node deleted, processing expiration [C3S31,20020,1326987900492]
2012-01-19 11:10:35,850 DEBUG org.apache.hadoop.hbase.master.ServerManager: Added=C3S31,20020,1326987900492
to dead servers, submitted shutdown handler to be executed, root=true, meta=false
2012-01-19 11:10:35,851 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Splitting logs for C3S31,20020,1326987900492
2012-01-19 11:12:37,003 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Archived
processed log hdfs://C3S31:9000/hbase/.logs/C3S31,20020,1326987900492/C3S31%3A20020.1326987901100
to hdfs://C3S31:9000/hbase/.oldlogs/C3S31%3A20020.1326987901100
2012-01-19 11:12:37,004 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: hlog file
splitting completed in 1137 ms for hdfs://C3S31:9000/hbase/.logs/C3S31,20020,1326987900492
2012-01-19 11:12:37,004 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Server C3S31,20020,1326987900492 was carrying ROOT. Trying to assign.
2012-01-19 11:12:37,382 INFO org.apache.hadoop.hbase.master.HMaster: .META. assigned=2, rit=false,
2012-01-19 11:12:37,382 INFO org.apache.hadoop.hbase.master.HMaster: Master startup proceeding:
master failover
2012-01-19 11:12:37,383 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
1028785192 on serverName=C3S32,20020,1326987400439, load=(requests=0, regions=0, usedHeap=0,
2012-01-19 11:12:37,384 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
70236052 on serverName=C3S32,20020,1326987400439, load=(requests=0, regions=0, usedHeap=0,
2012-01-19 11:12:37,393 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Reassigning 0 region(s) that C3S31,20020,1326987900492 was carrying (skipping 0 regions(s)
that are already in transition)
2012-01-19 11:12:37,393 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler:
Finished processing of shutdown of C3S31,20020,1326987900492
>From above log, we can see that .META. was assigned to C3S32 and log splitting was for

So was there data loss ?

> Concurrent processing of processFaileOver and ServerShutdownHandler may cause region
to be assigned before log splitting is completed, causing data loss
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>                 Key: HBASE-5179
>                 URL: https://issues.apache.org/jira/browse/HBASE-5179
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.90.2
>            Reporter: chunhui shen
>            Assignee: chunhui shen
>            Priority: Critical
>             Fix For: 0.92.0, 0.94.0, 0.90.6
>         Attachments: 5179-90.txt, 5179-90v10.patch, 5179-90v11.patch, 5179-90v12.patch,
5179-90v2.patch, 5179-90v3.patch, 5179-90v4.patch, 5179-90v5.patch, 5179-90v6.patch, 5179-90v7.patch,
5179-90v8.patch, 5179-90v9.patch, 5179-v11-92.txt, 5179-v11.txt, 5179-v2.txt, 5179-v3.txt,
5179-v4.txt, Errorlog, hbase-5179.patch, hbase-5179v10.patch, hbase-5179v12.patch, hbase-5179v5.patch,
hbase-5179v6.patch, hbase-5179v7.patch, hbase-5179v8.patch, hbase-5179v9.patch
> If master's processing its failover and ServerShutdownHandler's processing happen concurrently,
it may appear following  case.
> 1.master completed splitLogAfterStartup()
> 2.RegionserverA restarts, and ServerShutdownHandler is processing.
> 3.master starts to rebuildUserRegions, and RegionserverA is considered as dead server.
> 4.master starts to assign regions of RegionserverA because it is a dead server by step3.
> However, when doing step4(assigning region), ServerShutdownHandler may be doing split
log, Therefore, it may cause data loss.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message