hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Himanshu Vashishtha (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-9110) Meta region edits not recovered while migrating to 0.96.0
Date Tue, 27 Aug 2013 22:45:53 GMT

    [ https://issues.apache.org/jira/browse/HBASE-9110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13751828#comment-13751828
] 

Himanshu Vashishtha commented on HBASE-9110:
--------------------------------------------

Yeah, hbck thing is a bit hacky (when to run, when not to)

So, the root cause was .META. (i.e., hbase:meta) was opening up earlier than splitting to
happen. Attached patch ensures that we hold on opening any region unless all the log-split
is done. This will happen only when starting first time post-upgrade, and makes sure that
we are honoring all the previous edits. 
I tested it with creating tables, inserting data into a 0.94.x cluster and post migration,
data was intact.

Re: comment on HBASE-9278 what happens if we are migrating from 0.94 with separate meta log
enabled.
bq.  If it is a 0.94 install w/ the special meta WAL, the .META. -> hbase:meta fix will
work for this case too.

I checked (and tested) the separate .META. wal in 0.94. If we migrate from a 0.94.x (with
separate meta WAL), then the .meta wals are not splitted AT ALL while assigning meta, and
they are ignored by the subsequent log splitting process. 
It is because in 0.96, HMaster checks meta server znode (which doesn't point to any regionserver).
If there is no entry in that znode, it assigns meta region without doing any splitting (it
doesn't know which server had meta earlier).
{code}

if (!rit && !metaRegionLocation) {
      ServerName currentMetaServer = this.catalogTracker.getMetaLocation();
      if (currentMetaServer != null) {
        beingExpired = expireIfOnline(currentMetaServer);
      }
      if (beingExpired) {
        splitMetaLogBeforeAssignment(currentMetaServer);
      }
      assignmentManager.assignMeta();
      // Make sure a .META. location is set.
      enableSSHandWaitForMeta();
{code}

I tested the above patch with separate meta wal property OFF. It needs some modifications
to handle that feature, but would be good to know what you think about it. Thanks.
                
> Meta region edits not recovered while migrating to 0.96.0
> ---------------------------------------------------------
>
>                 Key: HBASE-9110
>                 URL: https://issues.apache.org/jira/browse/HBASE-9110
>             Project: HBase
>          Issue Type: Sub-task
>          Components: migration
>    Affects Versions: 0.95.2, 0.94.10
>            Reporter: Himanshu Vashishtha
>            Priority: Critical
>             Fix For: 0.96.0
>
>         Attachments: HBase-9110-v0.patch
>
>
> I was doing the migration testing from 0.94.11-snapshot to 0.95.0, and faced this issue.
> 1) Do some edits in meta table (for eg, create a table).
> 2) Kill the cluster.
> (I used kill because we would be doing log splitting when upgrading anyway).
> 3) There is some dependency on WALs. Upgrade the bits to 0.95.2-snapshot. Start the cluster.
> Every thing comes up. I see log splitting happening as expected. But, the WAL-data for
meta table is missing.
> I could see recovered.edits file for meta created, and placed at the right location.
It is just that the new HMaster code tries to recover meta by looking at meta prefix in the
log name, and if it didn't find one, just opens the meta region. So, the recovered.edits file,
created afterwards, is not honored.
> Opening this jira to let folks give their opinions about how to tackle this migration
issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message