hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2812) Disable 'table' fails to complete frustrating my ability to test easily
Date Thu, 05 Aug 2010 17:43:17 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895749#action_12895749
] 

Jean-Daniel Cryans commented on HBASE-2812:
-------------------------------------------

Dave sent me the logs, and it really looks like a case of HBASE-2755:

{noformat}
2010-08-04 19:26:58,291 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding region
test_ReferralQueue,,1280975190155 to setClosing list
2010-08-04 19:26:58,565 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {server: 192.168.21.66:60020, regionname: .META.,,1, startKey: <>}
2010-08-04 19:26:58,654 INFO org.apache.hadoop.hbase.master.ServerManager: Processing MSG_REPORT_CLOSE:
test_ReferralQueue,,1280975190155 from hslave3,60020,1280951333279; 1 of 1
2010-08-04 19:26:58,654 DEBUG org.apache.hadoop.hbase.master.HMaster: Processing todo: ProcessRegionClose
of test_ReferralQueue,,1280975190155, true, reassign: false
2010-08-04 19:26:58,656 INFO org.apache.hadoop.hbase.master.ProcessRegionClose$1: region closed:
test_ReferralQueue,,1280975190155
2010-08-04 19:26:58,686 DEBUG org.apache.hadoop.hbase.master.BaseScanner: GET on test_ReferralQueue,,1280975190155
got different startcode than SCAN: sc=0, serverAddress=1280951333279
2010-08-04 19:26:58,686 DEBUG org.apache.hadoop.hbase.master.BaseScanner: Current assignment
of test_ReferralQueue,,1280975190155 is not valid;  serverAddress=, startCode=0 unknown.
{noformat}

Here the base scanner saw that that region was half unassigned (atomicity problem?) since
we don't see the message that "sa" is different.   It means that the base scanner reassigns
the region because it believes that:

 - the region isn't assigned to anyone
 - that it's not online
 - that it's not in transition

This means that we should also verify our assumption that the HRI.isOffline() is correct when
we read it. I think we can augment HBASE-2755's patch to do that, since doing the GET we already
read the new HRI. Making a patch.

> Disable 'table' fails to complete frustrating my ability to test easily
> -----------------------------------------------------------------------
>
>                 Key: HBASE-2812
>                 URL: https://issues.apache.org/jira/browse/HBASE-2812
>             Project: HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.20.6, 0.89.20100621
>         Environment: 0.89, non-distributed mode
>            Reporter: Sam Pullara
>             Fix For: 0.20.7, 0.90.0
>
>
> I see this in the client after it gives up:
> hbase(main):006:0> disable 'test_schema'
> ERROR: org.apache.hadoop.hbase.RegionException: Retries exhausted, it took too long to
wait for the table test_schema to be disabled.
> Here is some help for this command:
>           Disable the named table: e.g. "hbase> disable 't1'"
> and this in the server log, a set of about 5 reports it is closing per disable call:
> 2010-07-03 15:19:47,554 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
unserved regions
> 2010-07-03 15:19:47,554 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
regions currently being served
> 2010-07-03 15:19:47,555 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding
region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:47,576 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:47,576 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:48,567 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
unserved regions
> 2010-07-03 15:19:48,567 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
regions currently being served
> 2010-07-03 15:19:48,568 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding
region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:48,577 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:48,578 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:49,580 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
unserved regions
> 2010-07-03 15:19:49,580 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
regions currently being served
> 2010-07-03 15:19:49,581 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding
region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:50,580 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:50,581 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:50,592 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
unserved regions
> 2010-07-03 15:19:50,592 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
regions currently being served
> 2010-07-03 15:19:50,593 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding
region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:51,581 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:51,581 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:52,605 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
unserved regions
> 2010-07-03 15:19:52,605 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
regions currently being served
> 2010-07-03 15:19:52,606 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding
region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:52,703 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers,
0 dead, average load 3.0
> 2010-07-03 15:19:52,863 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey:
<>}
> 2010-07-03 15:19:52,867 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of 1 row(s) of meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052,
startKey: <>} complete
> 2010-07-03 15:19:53,585 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:53,585 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:53,671 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats:
Sizes: Total=1.6292953MB (1708440), Free=196.70822MB (206263512), Max=198.33751MB (207971952),
Counts: Blocks=2, Access=52, Hit=50, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.15384340286255%,
Miss Ratio=3.8461539894342422%, Evicted/Run=NaN
> 2010-07-03 15:19:54,617 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
unserved regions
> 2010-07-03 15:19:54,617 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
regions currently being served
> 2010-07-03 15:19:54,618 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding
region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:54,821 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey:
<>}
> 2010-07-03 15:19:54,825 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 2 row(s) of meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192,
startKey: <>} complete
> 2010-07-03 15:19:54,825 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META.
region(s) scanned
> 2010-07-03 15:19:55,589 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:55,589 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:58,629 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
unserved regions
> 2010-07-03 15:19:58,629 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Processing
regions currently being served
> 2010-07-03 15:19:58,630 DEBUG org.apache.hadoop.hbase.master.ChangeTableState: Adding
region test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081. to setClosing list
> 2010-07-03 15:19:59,595 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_CLOSE:
test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:19:59,595 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker:
MSG_REGION_CLOSE: test_schema,,1278195322074.65c77aedf2f2a08d161a188dd2dd5081.
> 2010-07-03 15:20:52,706 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers,
0 dead, average load 3.0
> 2010-07-03 15:20:52,866 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey:
<>}
> 2010-07-03 15:20:52,873 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of 1 row(s) of meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052,
startKey: <>} complete
> 2010-07-03 15:20:53,671 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats:
Sizes: Total=1.6292953MB (1708440), Free=196.70822MB (206263512), Max=198.33751MB (207971952),
Counts: Blocks=2, Access=54, Hit=52, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.29629850387573%,
Miss Ratio=3.7037037312984467%, Evicted/Run=NaN
> 2010-07-03 15:20:54,823 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey:
<>}
> 2010-07-03 15:20:54,828 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 2 row(s) of meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192,
startKey: <>} complete
> 2010-07-03 15:20:54,828 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META.
region(s) scanned
> 2010-07-03 15:21:52,709 INFO org.apache.hadoop.hbase.master.ServerManager: 1 region servers,
0 dead, average load 3.0
> 2010-07-03 15:21:52,869 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scanning meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052, startKey:
<>}
> 2010-07-03 15:21:52,876 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.rootScanner
scan of 1 row(s) of meta region {server: 192.168.2.1:54389, regionname: -ROOT-,,0.70236052,
startKey: <>} complete
> 2010-07-03 15:21:53,672 DEBUG org.apache.hadoop.hbase.io.hfile.LruBlockCache: Cache Stats:
Sizes: Total=1.6292953MB (1708440), Free=196.70822MB (206263512), Max=198.33751MB (207971952),
Counts: Blocks=2, Access=56, Hit=54, Miss=2, Evictions=0, Evicted=0, Ratios: Hit Ratio=96.42857313156128%,
Miss Ratio=3.57142873108387%, Evicted/Run=NaN
> 2010-07-03 15:21:54,826 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scanning meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192, startKey:
<>}
> 2010-07-03 15:21:54,830 INFO org.apache.hadoop.hbase.master.BaseScanner: RegionManager.metaScanner
scan of 2 row(s) of meta region {server: 192.168.2.1:54389, regionname: .META.,,1.1028785192,
startKey: <>} complete
> 2010-07-03 15:21:54,830 INFO org.apache.hadoop.hbase.master.BaseScanner: All 1 .META.
region(s) scanned

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message