hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-10349) Table became unusable after taking snapshot, restoring snapshot and dropping table
Date Wed, 15 Jan 2014 23:26:23 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872765#comment-13872765
] 

Ted Yu edited comment on HBASE-10349 at 1/15/14 11:25 PM:
----------------------------------------------------------

The master started balancing the single region of tablethree_mod after tablethree_mod was
deleted:
{code}
2014-01-15 09:41:42,395 INFO  [AM.ZK.Worker-pool2-t857] master.RegionStates: Offlined afc82d1ceabbaca36a504b83b65fc0c9
from hor15n07.gq1.ygridcore.net,60020,1389770493951
...
2014-01-15 09:41:43,751 DEBUG [MASTER_TABLE_OPERATIONS-hor15n05:60000-0] handler.DeleteTableHandler:
Removing 'tablethree_mod' descriptor.
2014-01-15 09:41:43,752 DEBUG [MASTER_TABLE_OPERATIONS-hor15n05:60000-0] handler.DeleteTableHandler:
Marking 'tablethree_mod' as deleted.
2014-01-15 09:41:43,768 DEBUG [MASTER_TABLE_OPERATIONS-hor15n05:60000-0] lock.ZKInterProcessLockBase:
Released /hbase/table-lock/tablethree_mod/write-master:
...
2014-01-15 09:41:44,643 INFO  [hor15n05.gq1.ygridcore.net,60000,1389770492644-BalancerChore]
master.HMaster: balance hri=tablethree_mod,,1389778226606.                    afc82d1ceabbaca36a504b83b65fc0c9.,
src=hor15n07.gq1.ygridcore.net,60020,1389770493951, dest=hor15n09.gq1.ygridcore.net,60020,1389770493998
...
2014-01-15 09:41:44,668 DEBUG [hor15n05.gq1.ygridcore.net,60000,1389770492644-BalancerChore]
master.AssignmentManager: Setting table tablethree_mod to ENABLED state.
...
2014-01-15 09:41:44,761 DEBUG [AM.ZK.Worker-pool2-t873] master.AssignmentManager: Found an
existing plan for tablethree_mod,,1389778226606.                                afc82d1ceabbaca36a504b83b65fc0c9.
destination server is hor15n09.gq1.ygridcore.net,60020,1389770493998 accepted as a dest server
= false
2014-01-15 09:41:44,761 DEBUG [AM.ZK.Worker-pool2-t873] master.AssignmentManager: No previous
transition plan found (or ignoring an existing plan) for tablethree_mod,,    1389778226606.afc82d1ceabbaca36a504b83b65fc0c9.;
generated random plan=hri=tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9.,
src=, dest=hor15n12.gq1.       ygridcore.net,60020,1389770493928; 5 (online=5, available=4)
available servers, forceNewPlan=true
...
2014-01-15 09:41:57,784 WARN  [MASTER_TABLE_OPERATIONS-hor15n05:60000-0] catalog.MetaReader:
No serialized HRegionInfo in keyvalues={tablethree_mod,,1389778226606.        afc82d1ceabbaca36a504b83b65fc0c9./info:seqnumDuringOpen/1389778905355/Put/vlen=8/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:server/      1389778905355/Put/vlen=32/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:serverstartcode/1389778905355/Put/vlen=8/mvcc=0}
{code}


was (Author: yuzhihong@gmail.com):
The master started balancing the single region of tablethree_mod after tablethree_mod was
deleted:
{code}
2014-01-15 09:41:43,751 DEBUG [MASTER_TABLE_OPERATIONS-hor15n05:60000-0] handler.DeleteTableHandler:
Removing 'tablethree_mod' descriptor.
2014-01-15 09:41:43,752 DEBUG [MASTER_TABLE_OPERATIONS-hor15n05:60000-0] handler.DeleteTableHandler:
Marking 'tablethree_mod' as deleted.
2014-01-15 09:41:43,768 DEBUG [MASTER_TABLE_OPERATIONS-hor15n05:60000-0] lock.ZKInterProcessLockBase:
Released /hbase/table-lock/tablethree_mod/write-master:
...
2014-01-15 09:41:44,643 INFO  [hor15n05.gq1.ygridcore.net,60000,1389770492644-BalancerChore]
master.HMaster: balance hri=tablethree_mod,,1389778226606.                    afc82d1ceabbaca36a504b83b65fc0c9.,
src=hor15n07.gq1.ygridcore.net,60020,1389770493951, dest=hor15n09.gq1.ygridcore.net,60020,1389770493998
...
2014-01-15 09:41:44,668 DEBUG [hor15n05.gq1.ygridcore.net,60000,1389770492644-BalancerChore]
master.AssignmentManager: Setting table tablethree_mod to ENABLED state.
...
2014-01-15 09:41:44,761 DEBUG [AM.ZK.Worker-pool2-t873] master.AssignmentManager: Found an
existing plan for tablethree_mod,,1389778226606.                                afc82d1ceabbaca36a504b83b65fc0c9.
destination server is hor15n09.gq1.ygridcore.net,60020,1389770493998 accepted as a dest server
= false
2014-01-15 09:41:44,761 DEBUG [AM.ZK.Worker-pool2-t873] master.AssignmentManager: No previous
transition plan found (or ignoring an existing plan) for tablethree_mod,,    1389778226606.afc82d1ceabbaca36a504b83b65fc0c9.;
generated random plan=hri=tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9.,
src=, dest=hor15n12.gq1.       ygridcore.net,60020,1389770493928; 5 (online=5, available=4)
available servers, forceNewPlan=true
...
2014-01-15 09:41:57,784 WARN  [MASTER_TABLE_OPERATIONS-hor15n05:60000-0] catalog.MetaReader:
No serialized HRegionInfo in keyvalues={tablethree_mod,,1389778226606.        afc82d1ceabbaca36a504b83b65fc0c9./info:seqnumDuringOpen/1389778905355/Put/vlen=8/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:server/      1389778905355/Put/vlen=32/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:serverstartcode/1389778905355/Put/vlen=8/mvcc=0}
{code}

> Table became unusable after taking snapshot, restoring snapshot and dropping table
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-10349
>                 URL: https://issues.apache.org/jira/browse/HBASE-10349
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
>         Attachments: 10349-output.log, hbase-hbase-master-hor15n05.gq1.ygridcore.net.log.tar.gz
>
>
> 0.98 was used.
> This was sequence of events:
> create 'tablethree_mod'
> snapshot 'tablethree_mod', 'snapshot_tablethree_mod'
> disable 'tablethree_mod'
> 2014-01-15 09:34:51,749   restore_snapshot 'snapshot_tablethree_mod'
> 2014-01-15 09:35:07,210   enable 'tablethree_mod'
> 2014-01-15 09:35:46,134   delete_snapshot 'snapshot_tablethree_mod'
> 2014-01-15 09:41:42,210   disable 'tablethree_mod'
> 2014-01-15 09:41:43,610   drop 'tablethree_mod'
> create 'tablethree_mod'
> For the last table creation request:
> {code} 
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'create 'tablethree_mod',
> {NAME => 'f1', VERSIONS => 3}
> ,
> {NAME => 'f2', VERSIONS => 3}
> ,
> {NAME => 'f3', VERSIONS => 3}
> '
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'exists 'tablethree_mod''
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod', '0', 'f1:q1',
'value-0', 10'
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod', '1', 'f1:q1',
'value-1', 20'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '2', 'f2:q2',
'value-2', 30'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '3', 'f3:q3',
'value-3', 40'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '4', 'f3:q3',
'value-4', 50'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO|Done writing commands to file. Will
execute them now.
> 2014-01-15 10:03:53,000|beaver.machine|INFO|RUNNING: /usr/lib/hbase/bin/hbase shell /grid/0/tmp/hwqe/artifacts/tmp-471142
> 2014-01-15 10:03:55,878|beaver.machine|INFO|2014-01-15 10:03:55,878 INFO [main] Configuration.deprecation:
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2014-01-15 10:03:57,283|beaver.machine|INFO|2014-01-15 10:03:57,283 WARN [main] conf.Configuration:
hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
> 2014-01-15 10:03:57,669|beaver.machine|INFO|2014-01-15 10:03:57,669 WARN [main] conf.Configuration:
hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
> 2014-01-15 10:03:57,720|beaver.machine|INFO|2014-01-15 10:03:57,720 WARN [main] conf.Configuration:
hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
> 2014-01-15 10:03:57,997|beaver.machine|INFO|
> 2014-01-15 10:03:57,997|beaver.machine|INFO|ERROR: Table already exists: tablethree_mod!
> 2014-01-15 10:03:57,997|beaver.machine|INFO|
> {code}
> This was an intermittent issue after using Snapshots, a table is not properly dropped
/ and not able to properly re-create with the same name. And a HRegion is empty or null Error
occurs. (When you try to drop the table it says it does not exist, and when you try to create
the table it says that it does already exist).
> {code}
> 2014-01-15 10:04:02,462|beaver.machine|INFO|ERROR: HRegionInfo was null or empty in hbase:meta,
row=keyvalues=
> {tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:seqnumDuringOpen/1389778905355/Put/vlen=8/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:server/1389778905355/Put/vlen=32/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:serverstartcode/1389778905355/Put/vlen=8/mvcc=0}

> {code}
> Thanks to Huned who discovered this issue.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message