hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: Table can't disable
Date Mon, 16 May 2011 19:21:29 GMT
On Sat, May 14, 2011 at 7:08 PM, Gaojinchao <gaojinchao@huawei.com> wrote:
> I try to make a patch and process it.


OK.  What are you trying to do in the patch?  (Turns out my issue was
different to what you see).



> I want to send a zk message again when close region is timeout.
> I try to reproduce and verify it. But it is difficulty.

Yes, this stuff is tricky.  Its not easy making a test.  Have you seen
TestZKBasedOpenCloseRegion and TestMasterFailover.  These do messing
about with zk state.  Perhaps they help?

> Do you give me some help and review it.
>

Did you change code in the below?  If so, can you send a patch only?
Or better, attach it to a JIRA?  Patch is good because it is the
difference between what is hbase and what you have added.  Otherwise,
it takes a bit of work figuring what you have added.

Thank you Gao,
St.Ack

> Thanks.
>
> case PENDING_CLOSE:
>                LOG.info("Region has been PENDING_CLOSE for too " +
>                    "long, running forced unassign again on region=" +
>                    regionInfo.getRegionNameAsString());
>                  try {
>                    // If the server got the RPC, it will transition the node
>                    // to CLOSING, so only do something here if no node exists
>                    if (!ZKUtil.watchAndCheckExists(watcher,
>                      ZKAssign.getNodeName(watcher, regionInfo.getEncodedName())))
{
>                      // Queue running of an unassign -- do actual unassign
>                      // outside of the regionsInTransition lock.
>                      unassigns.add(regionInfo);
>                    }
>                    else
>                    {
>                                                          
                     // I add some code
>                       try {
>                          String node = ZKAssign.getNodeName(watcher,
>                             regionInfo.getEncodedName());
>                          Stat stat = new Stat();
>                          RegionTransitionData data = ZKAssign.getDataNoWatch(watcher,
>                             node, stat);
>                          if (data == null) {
>                             LOG.warn("Data is null, node " + node + " no
longer exists");
>                             break;
>                          }
>                          if (data.getEventType() != EventType.RS_ZK_REGION_CLOSED)
{
>                             LOG.debug("Region has transitioned to CLOEING,
allowing " +
>                                  "watched event handlers to process");
>                             break;
>                          }
>
>                         //In this case, Region server has ClOSED
>                         data = new RegionTransitionData(
>                                 EventType.RS_ZK_REGION_CLOSED, regionInfo.getRegionName(),
>                                    master.getServerName());
>                         if (ZKUtil.setData(watcher, node, data.getBytes(),
>                                   stat.getVersion()+1 )) {
>
>                            // Node is now closed, let's trigger another
close handler
>                            LOG.info("Try to finish closed region=" +
>                               regionInfo.getRegionNameAsString() + "again"
);
>                         }
>
>                      } catch (KeeperException ke) {
>                         LOG.error("Unexpected ZK exception timing out PENDING_CLOSE
region",
>                            ke);
>                         break;
>                        }
>
>                     }
>                  } catch (NoNodeException e) {
>                    LOG.debug("Node no longer existed so not forcing another
" +
>                      "unassignment");
>                  } catch (KeeperException e) {
>                    LOG.warn("Unexpected ZK exception timing out a region "
+
>                      "close", e);
>                  }
>                  break;
>
> -----邮件原件-----
> 发件人: saint.ack@gmail.com [mailto:saint.ack@gmail.com] 代表 Stack
> 发送时间: 2011年5月10日 1:38
> 收件人: dev@hbase.apache.org
> 主题: Re: Table can't disable
>
> I'm looking into this Gao.  Something similar seems to have happened
> here over the w/e.  Let me look at our systems first and then I'l be
> back to you on this below.  Thanks,
> St.Ack
>
> On Sun, May 8, 2011 at 4:58 AM, Gaojinchao <gaojinchao@huawei.com> wrote:
>> Today I test Hbase verison 0.90.3
>> I seems like some bugs
>> 1, if node exists and node state is RS_ZK_REGION_CLOSED
>>  We should call ClosedRegionHandle to close region. In that case the reigon has
closed by region server
>>
>> case PENDING_CLOSE:
>>                LOG.info("Region has been PENDING_CLOSE for too " +
>>                    "long, running forced unassign again on region=" +
>>                    regionInfo.getRegionNameAsString());
>>                  try {
>>                    // If the server got the RPC, it will transition the
node
>>                    // to CLOSING, so only do something here if no node
exists
>>                    if (!ZKUtil.watchAndCheckExists(watcher,
>>                      ZKAssign.getNodeName(watcher, regionInfo.getEncodedName())))
{
>>                      // Queue running of an unassign -- do actual unassign
>>                      // outside of the regionsInTransition lock.
>>                      unassigns.add(regionInfo);
>>                    }
>>                                        else
>>                    {
>>                                                //It need
to process for node state is RS_ZK_REGION_CLOSED
>>                                        }
>>                  } catch (NoNodeException e) {
>>                    LOG.debug("Node no longer existed so not forcing another
" +
>>                      "unassignment");
>>                  } catch (KeeperException e) {
>>                    LOG.warn("Unexpected ZK exception timing out a region
" +
>>                      "close", e);
>>                  }
>> 2, other It seems like zk message losing or there is some bugs for disabling table
when region server is splitting region.
>> above is the logs. I need to keep digging.
>>
>> Shell logs:
>> hbase(main):003:0*
>> hbase(main):004:0* disable 'ufdr'
>> 11/05/08 18:20:40 DEBUG zookeeper.ZKUtil: hconnection-0x2fcd58283c0037 Set watcher
on existing znode /hbase/root-region-server
>> 11/05/08 18:20:40 DEBUG zookeeper.ZKUtil: hconnection-0x2fcd58283c0037 Retrieved
15 byte(s) of data from znode /hbase/root-region-server and set watcher; C4C5.site:60020
>> 11/05/08 18:53:19 DEBUG zookeeper.ZKUtil: hconnection-0x2fcd58283c0037 Retrieved
9 byte(s) of data from znode /hbase/table/ufdr; data=DISABLING
>>
>> ERROR: org.apache.hadoop.hbase.RegionException: Retries exhausted, it took too long
to wait for the table ufdr to be disabled.
>>
>>
>> Region server logs:
>>
>> 2011-05-08 17:42:44,862 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:42:45,468 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded
hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/value/3442771824694350714.8e9a3b05abe1c3a692999cf5e8dfd9dd,
isReference=true, isBulkLoadResult=false, seqid=627830, majorCompaction=false
>> 2011-05-08 17:42:45,471 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.; next sequenceid=627831
>> 2011-05-08 17:42:45,471 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Compaction requested for ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
because Region has references on open; priority=9, compaction queue size=40
>> 2011-05-08 17:42:45,476 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Added daughter
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. in region
.META.,,1, serverInfo=C4C4.site,60020,1304820199467
>> 2011-05-08 17:42:45,476 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Region split, META updated, and report to master. Parent=ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.,
new regions: ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.. Split
took 0sec
>> 2011-05-08 17:44:25,731 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Received close region: ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:48:11,066 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting
compaction on region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:48:11,067 INFO org.apache.hadoop.hbase.regionserver.Store: Started
compaction of 1 file(s) in cf=value, hasReferences=true, into hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/.tmp,
seqid=627830, totalSize=892.2m
>> 2011-05-08 17:48:11,067 DEBUG org.apache.hadoop.hbase.regionserver.Store: Compacting
hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/value/3442771824694350714.8e9a3b05abe1c3a692999cf5e8dfd9dd-hdfs://C4C1:9000/hbase/ufdr/8e9a3b05abe1c3a692999cf5e8dfd9dd/value/3442771824694350714-top,
keycount=2843998, bloomtype=NONE, size=892.2m
>> 2011-05-08 17:48:32,340 INFO org.apache.hadoop.hbase.regionserver.Store: Completed
compaction of 1 file(s), new file=hdfs://C4C1:9000/hbase/ufdr/4418fb197685a21f77e151e401cf8b66/value/4640085674257062204,
size=446.2m; total size for store is 446.2m
>> 2011-05-08 17:48:32,340 INFO org.apache.hadoop.hbase.regionserver.HRegion: completed
compaction on region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
after 21sec
>> 2011-05-08 17:52:46,626 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
Processing close of ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:46,627 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003
Creating unassigned node for 4418fb197685a21f77e151e401cf8b66 in a CLOSING state
>> 2011-05-08 17:52:46,986 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.: disabling
compactions & flushes
>> 2011-05-08 17:52:46,986 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates
disabled for region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:46,986 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:46,986 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003
Attempting to transition node 4418fb197685a21f77e151e401cf8b66 from RS_ZK_REGION_CLOSING to
RS_ZK_REGION_CLOSED
>> 2011-05-08 17:52:47,052 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003
Retrieved 125 byte(s) of data from znode /hbase/unassigned/4418fb197685a21f77e151e401cf8b66;
data=region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
server=C4C4.site,60020,1304820199467, state=RS_ZK_REGION_CLOSING
>> 2011-05-08 17:52:47,143 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:
regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003 Received ZooKeeper Event, type=NodeDataChanged,
state=SyncConnected, path=/hbase/unassigned/4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,143 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x12fcd58280c0003-0x12fcd58280c0003
Successfully transitioned node 4418fb197685a21f77e151e401cf8b66 from RS_ZK_REGION_CLOSING
to RS_ZK_REGION_CLOSED
>> 2011-05-08 17:52:47,143 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler:
Closed region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:47,696 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Received close region: ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:47,696 WARN org.apache.hadoop.hbase.regionserver.HRegionServer:
Received close for region we are not serving; 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,814 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Received close region: ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:47,814 WARN org.apache.hadoop.hbase.regionserver.HRegionServer:
Received close for region we are not serving; 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:48,167 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
Received close region: ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:52:48,167 WARN org.apache.hadoop.hbase.regionserver.HRegionServer:
Received close for region we are not serving; 4418fb197685a21f77e151e401cf8b66
>>
>>
>> Hmaster logs:
>> 2011-05-08 17:42:45,514 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:43:37,599 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:43:45,525 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:44:25,745 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Sent
CLOSE to serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123, usedHeap=4097,
maxHeap=8175) for region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:44:45,530 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:45:45,542 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:46:45,303 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:46:45,538 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:47:45,548 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:48:45,545 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:49:46,108 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:50:46,105 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:51:46,117 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:52:46,112 INFO org.apache.hadoop.hbase.master.ServerManager: Received
REGION_SPLIT: ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.:
Daughters; ufdr,2011050812#8613817306227#0516,1304847764729.5e4bca85c33fa6605ffc9a5c2eb94e62.,
ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66. from C4C4.site,60020,1304820199467
>> 2011-05-08 17:52:47,309 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x22fcd582836003d
Retrieved 125 byte(s) of data from znode /hbase/unassigned/4418fb197685a21f77e151e401cf8b66
and set watcher; region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
server=C4C4.site,60020,1304820199467, state=RS_ZK_REGION_CLOSED
>> 2011-05-08 17:52:47,388 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
new unassigned node: /hbase/unassigned/4418fb197685a21f77e151e401cf8b66 (region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
server=C4C4.site,60020,1304820199467, state=RS_ZK_REGION_CLOSED)
>> 2011-05-08 17:52:47,388 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling
transition=RS_ZK_REGION_CLOSED, server=C4C4.site,60020,1304820199467, region=4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,388 WARN org.apache.hadoop.hbase.master.AssignmentManager: Received
CLOSED for region 4418fb197685a21f77e151e401cf8b66 from server C4C4.site,60020,1304820199467
but region was in  the state null and not in expected PENDING_CLOSE or CLOSING states
>> 2011-05-08 17:52:47,397 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,398 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:47,400 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:47,501 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,501 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,702 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempting
to unassign region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:47,704 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Server
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123, usedHeap=4097, maxHeap=8175)
returned org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException:
Received close for ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but we are not serving it for 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,816 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,819 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:47,820 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,820 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,820 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:47,820 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:47,821 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,821 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Server
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123, usedHeap=4097, maxHeap=8175)
returned org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException:
Received close for ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but we are not serving it for 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:47,821 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:47,822 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting
4418fb197685a21f77e151e401cf8b66 on serverName=C4C4.site,60020,1304820199467, load=(requests=0,
regions=123, usedHeap=4097, maxHeap=8175)
>> 2011-05-08 17:52:48,172 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:48,173 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:48,174 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempting
to unassign region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,174 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:48,175 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Server
serverName=C4C4.site,60020,1304820199467, load=(requests=0, regions=123, usedHeap=4097, maxHeap=8175)
returned org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException:
Received close for ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but we are not serving it for 4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:52:48,175 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:48,176 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
(offlining)
>> 2011-05-08 17:52:48,176 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempting
to unassign region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,177 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempting
to unassign region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,178 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempting
to unassign region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,178 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempting
to unassign region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,178 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Attempting
to unassign region ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
but it is already in transition (PENDING_CLOSE)
>> 2011-05-08 17:52:48,690 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted
daughter reference ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.,
qualifier=splitB, from parent ufdr,2011050812#8613817306227#0516,1304845660567.8e9a3b05abe1c3a692999cf5e8dfd9dd.
>> 2011-05-08 17:55:51,787 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
state=PENDING_CLOSE, ts=1304848368173
>> 2011-05-08 17:55:51,787 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been PENDING_CLOSE for too long, running forced unassign again on region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:55:51,787 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x22fcd582836003d
Set watcher on existing znode /hbase/unassigned/4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:56:01,774 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
state=PENDING_CLOSE, ts=1304848368173
>> 2011-05-08 17:56:01,774 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been PENDING_CLOSE for too long, running forced unassign again on region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:56:01,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x22fcd582836003d
Set watcher on existing znode /hbase/unassigned/4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:56:11,774 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
state=PENDING_CLOSE, ts=1304848368173
>> 2011-05-08 17:56:11,774 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region
has been PENDING_CLOSE for too long, running forced unassign again on region=ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
>> 2011-05-08 17:56:11,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: master:60000-0x22fcd582836003d
Set watcher on existing znode /hbase/unassigned/4418fb197685a21f77e151e401cf8b66
>> 2011-05-08 17:56:21,775 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions
in transition timed out:  ufdr,2011050812#8613817398167#4032,1304847764729.4418fb197685a21f77e151e401cf8b66.
state=PENDING_CLOSE, ts=1304848368173
>> 2011-05-08 17:5
>>
>>
>

Mime
View raw message