hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Yu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-10349) Table became unusable when master balanced its region after table was dropped
Date Thu, 16 Jan 2014 01:42:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872891#comment-13872891
] 

Ted Yu edited comment on HBASE-10349 at 1/16/14 1:42 AM:
---------------------------------------------------------

In the master log:
{code}
2014-01-15 09:41:42,395 INFO  [AM.ZK.Worker-pool2-t857] master.RegionStates: Offlined afc82d1ceabbaca36a504b83b65fc0c9
from hor15n07.gq1.ygridcore.net,60020,1389770493951
{code}
This should correspond to this line in regionOffline():
{code}
      LOG.info("Offlined " + hri.getShortNameToLog() + " from " + oldServerName);
        Set<HRegionInfo> oldRegions = serverHoldings.get(oldServerName);
        oldRegions.remove(hri);
{code}
But if hri was removed from oldRegions, I don't see why master would issue balance to the
same region: afc82d1ceabbaca36a504b83b65fc0c9


was (Author: yuzhihong@gmail.com):
>From namenode log I don't see the deletion corresponding to what Matteo posted @ 2014-01-15
09:41:43,639 :
{code}
2014-01-15 09:41:42,322 INFO  hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(2873))
- BLOCK* allocateBlock: /apps/hbase/data/data/default/tablethree_mod/          afc82d1ceabbaca36a504b83b65fc0c9/.tmp/2281cdbf37c54ab3b8dcc57069cae993.
BP-1099602576-68.142.247.161-1389770223091 blk_1073743631_2807{blockUCState=UNDER_CONSTRUCTION,
   primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[68.142.246.154:50010|RBW], ReplicaUnderConstruction[68.142.246.158:50010|RBW],
ReplicaUnderConstruction[68.142.246.159:50010|RBW]]}
...
2014-01-15 09:41:42,344 INFO  hdfs.StateChange (FSNamesystem.java:completeFile(2814)) - DIR*
completeFile: /apps/hbase/data/data/default/tablethree_mod/                   afc82d1ceabbaca36a504b83b65fc0c9/.tmp/2281cdbf37c54ab3b8dcc57069cae993
is closed by DFSClient_hb_rs_hor15n07.gq1.ygridcore.net,60020,1389770493951_719601563_28
...
2014-01-15 09:41:44,868 INFO  hdfs.StateChange (FSNamesystem.java:saveAllocatedBlock(2873))
- BLOCK* allocateBlock: /apps/hbase/data/data/default/tablethree_mod/          afc82d1ceabbaca36a504b83b65fc0c9/.tmp/.regioninfo.
BP-1099602576-68.142.247.161-1389770223091 blk_1073743636_2812{blockUCState=UNDER_CONSTRUCTION,
primaryNodeIndex=-1,    replicas=[ReplicaUnderConstruction[68.142.246.159:50010|RBW], ReplicaUnderConstruction[68.142.246.156:50010|RBW],
ReplicaUnderConstruction[68.142.246.154:50010|RBW]]}
{code}

> Table became unusable when master balanced its region after table was dropped
> -----------------------------------------------------------------------------
>
>                 Key: HBASE-10349
>                 URL: https://issues.apache.org/jira/browse/HBASE-10349
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Yu
>         Attachments: 10349-hadoop-hdfs-namenode-hor11n14.gq1.ygridcore.net.zip, 10349-output.log,
hbase-hbase-master-hor15n05.gq1.ygridcore.net.log.tar.gz
>
>
> 0.98 was used.
> This was sequence of events:
> create 'tablethree_mod'
> snapshot 'tablethree_mod', 'snapshot_tablethree_mod'
> disable 'tablethree_mod'
> 2014-01-15 09:34:51,749   restore_snapshot 'snapshot_tablethree_mod'
> 2014-01-15 09:35:07,210   enable 'tablethree_mod'
> 2014-01-15 09:35:46,134   delete_snapshot 'snapshot_tablethree_mod'
> 2014-01-15 09:41:42,210   disable 'tablethree_mod'
> 2014-01-15 09:41:43,610   drop 'tablethree_mod'
> create 'tablethree_mod'
> For the last table creation request:
> {code} 
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'create 'tablethree_mod',
> {NAME => 'f1', VERSIONS => 3}
> ,
> {NAME => 'f2', VERSIONS => 3}
> ,
> {NAME => 'f3', VERSIONS => 3}
> '
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'exists 'tablethree_mod''
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod', '0', 'f1:q1',
'value-0', 10'
> 2014-01-15 10:03:52,999|beaver.component.hbase|INFO| 'put 'tablethree_mod', '1', 'f1:q1',
'value-1', 20'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '2', 'f2:q2',
'value-2', 30'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '3', 'f3:q3',
'value-3', 40'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO| 'put 'tablethree_mod', '4', 'f3:q3',
'value-4', 50'
> 2014-01-15 10:03:53,000|beaver.component.hbase|INFO|Done writing commands to file. Will
execute them now.
> 2014-01-15 10:03:53,000|beaver.machine|INFO|RUNNING: /usr/lib/hbase/bin/hbase shell /grid/0/tmp/hwqe/artifacts/tmp-471142
> 2014-01-15 10:03:55,878|beaver.machine|INFO|2014-01-15 10:03:55,878 INFO [main] Configuration.deprecation:
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2014-01-15 10:03:57,283|beaver.machine|INFO|2014-01-15 10:03:57,283 WARN [main] conf.Configuration:
hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
> 2014-01-15 10:03:57,669|beaver.machine|INFO|2014-01-15 10:03:57,669 WARN [main] conf.Configuration:
hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
> 2014-01-15 10:03:57,720|beaver.machine|INFO|2014-01-15 10:03:57,720 WARN [main] conf.Configuration:
hbase-site.xml:an attempt to override final parameter: dfs.support.append; Ignoring.
> 2014-01-15 10:03:57,997|beaver.machine|INFO|
> 2014-01-15 10:03:57,997|beaver.machine|INFO|ERROR: Table already exists: tablethree_mod!
> 2014-01-15 10:03:57,997|beaver.machine|INFO|
> {code}
> This was an intermittent issue after using Snapshots, a table is not properly dropped
/ and not able to properly re-create with the same name. And a HRegion is empty or null Error
occurs. (When you try to drop the table it says it does not exist, and when you try to create
the table it says that it does already exist).
> {code}
> 2014-01-15 10:04:02,462|beaver.machine|INFO|ERROR: HRegionInfo was null or empty in hbase:meta,
row=keyvalues=
> {tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:seqnumDuringOpen/1389778905355/Put/vlen=8/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:server/1389778905355/Put/vlen=32/mvcc=0,
tablethree_mod,,1389778226606.afc82d1ceabbaca36a504b83b65fc0c9./info:serverstartcode/1389778905355/Put/vlen=8/mvcc=0}

> {code}
> Thanks to Huned who discovered this issue.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message