hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: NotReplicatedYetException when gracefully shutting down a region server.
Date Thu, 05 Jan 2012 17:54:45 GMT
That's very interesting, did you look at that region server's log to
see what was going on when creating that file? Normally if a region
server encounters a FATAL fs error (like not being able to create a
file) it will abort itself.

J-D

On Tue, Jan 3, 2012 at 4:14 PM, Alok Singh <alok@urbanairship.com> wrote:
> J-D,
>
> On the regionserver, I see these lines:
> -------------
> 2011-12-28 18:56:37,821 - INFO
> [regionserver7040.cacheFlusher:Store@494] - Renaming flushed file at
> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/745069841898422651
> to hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/user/8857592033950378228
> 2011-12-28 18:56:37,867 - INFO
> [regionserver7040.cacheFlusher:StoreFile$Reader@1027] - Loaded row
> bloom filter metadata for
> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/user/8857592033950378228
> 2011-12-28 18:56:37,867 - INFO
> [regionserver7040.cacheFlusher:Store@504] - Added
> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/user/8857592033950378228,
> entries=15990, sequenceid=1213107064, memsize=3.8m, filesize=200.1k
> 2011-12-28 19:22:57,674 - INFO  [PRI IPC Server handler 9 on
> 7040:HRegionServer@2142] - Received close region:
> rich_push.user,4e3b951f2325b4695800679d,1317431456933.0fd5a7b472fb092a4470e9505c3a421a.
> 2011-12-28 19:22:57,702 - INFO
> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:StoreFile$Writer@868]
> - Bloom added to HFile
> (hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/8759993386901964725):
> 32.0, 14/26 (54%)
> 2011-12-28 19:22:57,776 - INFO
> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:Store@494] -
> Renaming flushed file at
> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/8759993386901964725
> to hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/user/4476093570994008279
> 2011-12-28 19:22:57,893 - INFO
> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:StoreFile$Reader@1027]
> - Loaded row bloom filter metadata for
> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/user/4476093570994008279
> 2011-12-28 19:22:57,894 - INFO
> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:Store@504] - Added
> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/user/4476093570994008279,
> entries=210, sequenceid=1213258636, memsize=50.8k, filesize=3.6k
> 2011-12-28 19:22:57,896 - INFO
> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:HRegion@554] -
> Closed rich_push.user,4e3b951f2325b4695800679d,1317431456933.0fd5a7b472fb092a4470e9505c3a421a.
> 2011-12-29 17:14:59,900 - ERROR
> [Thread-15:DFSClient$LeaseChecker@1135] - Exception closing file
> /richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> : org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> ----------------------------------------
> We waited overnight after moving the regions from the regionserver
> before shutting it down. That is why the timestamp between the
> RS_CLOSE_REGION and ERROR are off by 22 hours.
>
> But, looking on the namenode, I found some interesting log entries.
> Note the timestamp on these entries, they are over a week before the
> regionserver errors occurred.
> -------namenode-logs---------
> rm=null
> 2011-12-20 03:47:41,208 - INFO  [IPC Server handler 8 on
> 7080:FSNamesystem@125] -
> ugi=hbaseregion ip=/10.129.1.230        cmd=create      src=/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
      dst=null        perm=hbaseregion:hadoop:rw-rw-rw-
> 2011-12-20 03:47:41,240 - INFO  [IPC Server handler 60 on
> 7080:FSNamesystem@1646] - BLOCK* NameSystem.allocateBlock:
> /richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423.
> blk_4226819991054427111_66443
> 2011-12-20 03:48:10,096 - INFO  [IPC Server handler 40 on
> 7080:FSNamesystem@1646] - BLOCK* NameSystem.allocateBlock:
> /richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423.
> blk_51125950084620321_66443
> 2011-12-20 03:48:38,808 - INFO  [IPC Server handler 1 on
> 7080:Server$Handler@1423] - IPC Server handler 1 on 7080, call
> addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423,
> DFSClient_201969766, null) from 10.129.1.230:54052: error:
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> 2011-12-20 03:48:39,245 - INFO  [IPC Server handler 22 on
> 7080:Server$Handler@1423] - IPC Server handler 22 on 7080, call
> addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423,
> DFSClient_201969766, null) from 10.129.1.230:54052: error:
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> 2011-12-20 03:48:40,055 - INFO  [IPC Server handler 42 on
> 7080:Server$Handler@1423] - IPC Server handler 42 on 7080, call
> addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423,
> DFSClient_201969766, null) from 10.129.1.230:54052: error:
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> 2011-12-20 03:48:41,665 - INFO  [IPC Server handler 63 on
> 7080:Server$Handler@1423] - IPC Server handler 63 on 7080, call
> addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423,
> DFSClient_201969766, null) from 10.129.1.230:54052: error:
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> 2011-12-20 03:48:44,875 - INFO  [IPC Server handler 26 on
> 7080:Server$Handler@1423] - IPC Server handler 26 on 7080, call
> addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423,
> DFSClient_201969766, null) from 10.129.1.230:54052: error:
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> 2011-12-20 03:48:51,285 - INFO  [IPC Server handler 11 on
> 7080:Server$Handler@1423] - IPC Server handler 11 on 7080, call
> addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423,
> DFSClient_201969766, null) from 10.129.1.230:54052: error:
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
> 2011-12-20 06:34:21,299 - INFO  [IPC Server handler 10 on
> 7080:FSNamesystem@125] -
> ugi=hbaseregion ip=/10.129.1.230        cmd=listStatus  src=/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/attribute
     dst=null        perm=null
> 2011-12-20 06:34:21,303 - INFO  [IPC Server handler 19 on
> 7080:FSNamesystem@125] -
> ugi=hbaseregion ip=/10.129.1.230        cmd=listStatus  src=/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/user
  dst=null        perm=null
> ....
> -----namenode-logs------------------
>
> Thanks for helping me figure out this error.
>
> Alok
>
> On Tue, Jan 3, 2012 at 2:06 PM, Jean-Daniel Cryans <jdcryans@apache.org> wrote:
>> Since you did the region mover thing then all the region should be off
>> from that region server, so data loss is highly unlikely.
>>
>> My theory would be that a compaction was underway and got cancelled
>> when the region server was told to close the region, but the
>> DFSOutputStream thread hasn't been cleanup yet. Probably the file was
>> even already deleted and the data is still safe in the un-compacted
>> files.
>>
>> Ways to confirm this:
>>
>>  - Look in the region server log in the preceding lines what happens
>> to that region (named 0fd5a7b472fb092a4470e9505c3a421a) when it's
>> closing. Do you see a compaction being cancelled?
>>  - Look in the Namenode log for that file, is it being asked to delete
>> it or something before it sends a NotReplicatedYetException?
>>
>> Hope this helps,
>>
>> J-D
>>
>> On Thu, Dec 29, 2011 at 3:11 PM, Alok Singh <alok@urbanairship.com> wrote:
>>> When attempting to gracefully shutdown a regionserver, I saw a couple
>>> of NotReplicatedYet exceptions in the logs (below). Can't find the
>>> file that is causing this exception in on the HDFS filesystem. Have we
>>> potentially lost the data, or is this exception benign?
>>>
>>>
>>> Alok
>>>
>>> hbase: 0.90.3
>>> hadoop: 0.20.2-cdh3u0
>>>
>>> Gracefull shutdown process:
>>> hbase(main):001:0> balance_switch false
>>>
>>> hbase org.jruby.Main current/bin/region_mover.rb unload <IP>
>>>
>>> After the region count for the regionserver is 0, we kill the
>>> regionserver process.
>>>
>>> ------------------Log-------------------------
>>> 2011-12-29 17:11:03,768 - INFO  [PRI IPC Server handler 6 on
>>> 7040:HRegionServer@2142] - Received close region:
>>> rich_push.user_device,ujoG9VuxSbKTLOQalFdJDA.c9QrCxywSJWvSfYhDGL1XA,1324711834779.01c7894eea30fcd6713b159ca1645da5.
>>> 2011-12-29 17:11:03,787 - WARN
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:FsPermission@220] -
>>> dfs.umask configuration key is deprecated. Convert to dfs.umaskmode,
>>> using octal or symbolic umask specifications.
>>> 2011-12-29 17:11:03,803 - WARN
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:FsPermission@220] -
>>> dfs.umask configuration key is deprecated. Convert to dfs.umaskmode,
>>> using octal or symbolic umask specifications.
>>> 2011-12-29 17:11:03,911 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:StoreFile$Writer@868]
>>> - Bloom added to HFile
>>> (hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_device/01c7894eea30fcd6713b159ca1645da5/.tmp/6825595110476022990):
>>> 7.1k, 5225/5276 (99%)
>>> 2011-12-29 17:11:03,944 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:Store@494] -
>>> Renaming flushed file at
>>> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_device/01c7894eea30fcd6713b159ca1645da5/.tmp/6825595110476022990
>>> to hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_device/01c7894eea30fcd6713b159ca1645da5/device/1580164973134104965
>>> 2011-12-29 17:11:03,961 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:StoreFile$Reader@1027]
>>> - Loaded col bloom filter metadata for
>>> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_device/01c7894eea30fcd6713b159ca1645da5/device/1580164973134104965
>>> 2011-12-29 17:11:03,961 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:Store@504] - Added
>>> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_device/01c7894eea30fcd6713b159ca1645da5/device/1580164973134104965,
>>> entries=5276, sequenceid=1219931822, memsize=2.0m, filesize=384.4k
>>> 2011-12-29 17:11:03,964 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:HRegion@554] -
>>> Closed rich_push.user_device,ujoG9VuxSbKTLOQalFdJDA.c9QrCxywSJWvSfYhDGL1XA,1324711834779.01c7894eea30fcd6713b159ca1645da5.
>>> 2011-12-29 17:11:05,706 - INFO  [PRI IPC Server handler 7 on
>>> 7040:HRegionServer@2142] - Received close region:
>>> rich_push.user_udid,7sDqU74XQN6x1LWFPsmomQ.4e77ba299bf7e812c5002186,1325081217300.6f4cf99ff967f159fc68cd6c05001e9a.
>>> 2011-12-29 17:11:05,716 - WARN
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:FsPermission@220] -
>>> dfs.umask configuration key is deprecated. Convert to dfs.umaskmode,
>>> using octal or symbolic umask specifications.
>>> 2011-12-29 17:11:05,735 - WARN
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:FsPermission@220] -
>>> dfs.umask configuration key is deprecated. Convert to dfs.umaskmode,
>>> using octal or symbolic umask specifications.
>>> 2011-12-29 17:11:05,808 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:StoreFile$Writer@868]
>>> - Bloom added to HFile
>>> (hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid/6f4cf99ff967f159fc68cd6c05001e9a/.tmp/1207401926740467097):
>>> 6.5k, 4727/4737 (100%)
>>> 2011-12-29 17:11:05,834 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:Store@494] -
>>> Renaming flushed file at
>>> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid/6f4cf99ff967f159fc68cd6c05001e9a/.tmp/1207401926740467097
>>> to hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid/6f4cf99ff967f159fc68cd6c05001e9a/udid/8492436804961336492
>>> 2011-12-29 17:11:05,852 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:StoreFile$Reader@1027]
>>> - Loaded col bloom filter metadata for
>>> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid/6f4cf99ff967f159fc68cd6c05001e9a/udid/8492436804961336492
>>> 2011-12-29 17:11:05,852 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:Store@504] - Added
>>> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid/6f4cf99ff967f159fc68cd6c05001e9a/udid/8492436804961336492,
>>> entries=4737, sequenceid=1219931829, memsize=1.4m, filesize=246.5k
>>> 2011-12-29 17:11:05,856 - INFO
>>> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:HRegion@554] -
>>> Closed rich_push.user_udid,7sDqU74XQN6x1LWFPsmomQ.4e77ba299bf7e812c5002186,1325081217300.6f4cf99ff967f159fc68cd6c05001e9a.
>>> 2011-12-29 17:14:58,942 - INFO
>>> [Shutdownhook:regionserver7040:ShutdownHook$ShutdownHookThread@100] -
>>> Shutdown hook starting; hbase.shutdown.hook=true;
>>> fsShutdownHook=Thread[Thread-15,5,main]
>>> 2011-12-29 17:14:58,942 - INFO
>>> [Shutdownhook:regionserver7040:HRegionServer@1342] - STOPPED: Shutdown
>>> hook
>>> 2011-12-29 17:14:59,476 - INFO  [regionserver7040:HBaseServer@1234] -
>>> Stopping server on 7040
>>> 2011-12-29 17:14:59,476 - INFO  [PRI IPC Server handler 2 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 2 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,476 - INFO  [IPC Server handler 9 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 9 on 7040: exiting
>>> 2011-12-29 17:14:59,476 - INFO  [PRI IPC Server handler 4 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 4 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,476 - INFO  [IPC Server handler 4 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 4 on 7040: exiting
>>> 2011-12-29 17:14:59,477 - INFO  [IPC Server handler 5 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 5 on 7040: exiting
>>> 2011-12-29 17:14:59,477 - INFO  [IPC Server handler 6 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 6 on 7040: exiting
>>> 2011-12-29 17:14:59,477 - INFO  [IPC Server handler 7 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 7 on 7040: exiting
>>> 2011-12-29 17:14:59,477 - INFO  [IPC Server handler 8 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 8 on 7040: exiting
>>> 2011-12-29 17:14:59,477 - INFO  [PRI IPC Server handler 0 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 0 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,478 - INFO  [PRI IPC Server handler 1 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 1 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,478 - INFO  [PRI IPC Server handler 3 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 3 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,478 - INFO  [PRI IPC Server handler 5 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 5 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,478 - INFO  [PRI IPC Server handler 6 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 6 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,478 - INFO  [PRI IPC Server handler 7 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 7 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,478 - INFO  [PRI IPC Server handler 9 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 9 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,476 - INFO  [IPC Server handler 0 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 0 on 7040: exiting
>>> 2011-12-29 17:14:59,480 - INFO  [IPC Server
>>> Responder:HBaseServer$Responder@649] - Stopping IPC Server Responder
>>> 2011-12-29 17:14:59,479 - INFO  [IPC Server listener on
>>> 7040:HBaseServer$Listener@450] - Stopping IPC Server listener on 7040
>>> 2011-12-29 17:14:59,477 - INFO  [regionserver7040:HRegionServer@636] -
>>> Stopping infoServer
>>> 2011-12-29 17:14:59,476 - INFO  [PRI IPC Server handler 8 on
>>> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 8 on 7040:
>>> exiting
>>> 2011-12-29 17:14:59,476 - INFO  [IPC Server handler 3 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 3 on 7040: exiting
>>> 2011-12-29 17:14:59,476 - INFO  [IPC Server handler 2 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 2 on 7040: exiting
>>> 2011-12-29 17:14:59,476 - INFO  [IPC Server handler 1 on
>>> 7040:HBaseServer$Handler@1104] - IPC Server handler 1 on 7040: exiting
>>> 2011-12-29 17:14:59,481 - INFO  [regionserver7040:Slf4jLog@67] -
>>> Stopped SelectChannelConnector@0.0.0.0:7041
>>> 2011-12-29 17:14:59,517 - INFO
>>> [regionserver7040.logRoller:LogRoller@114] - LogRoller exiting.
>>> 2011-12-29 17:14:59,517 - INFO
>>> [regionserver7040.cacheFlusher:MemStoreFlusher@266] -
>>> regionserver7040.cacheFlusher exiting
>>> 2011-12-29 17:14:59,517 - INFO
>>> [regionserver7040.logSyncer:HLog$LogSyncer@966] -
>>> regionserver7040.logSyncer exiting
>>> 2011-12-29 17:14:59,518 - INFO
>>> [regionserver7040.majorCompactionChecker:Chore@79] -
>>> regionserver7040.majorCompactionChecker exiting
>>> 2011-12-29 17:14:59,518 - INFO
>>> [regionserver7040.compactor:CompactSplitThread@113] -
>>> regionserver7040.compactor exiting
>>> 2011-12-29 17:14:59,891 - INFO  [regionserver7040:HRegionServer@668] -
>>> stopping server at: 10.129.1.230,7040,1323990853621
>>> 2011-12-29 17:14:59,891 - INFO  [regionserver7040:Leases@124] -
>>> regionserver7040 closing leases
>>> 2011-12-29 17:14:59,891 - INFO  [regionserver7040:Leases@131] -
>>> regionserver7040 closed leases
>>> 2011-12-29 17:14:59,891 - INFO
>>> [regionserver7040:HConnectionManager$HConnectionImplementation@1067] -
>>> Closed zookeeper sessionid=0x4343f266cbc0012
>>> 2011-12-29 17:14:59,894 - INFO  [regionserver7040:ZooKeeper@538] -
>>> Session: 0x4343f266cbc0012 closed
>>> 2011-12-29 17:14:59,894 - INFO
>>> [main-EventThread:ClientCnxn$EventThread@520] - EventThread shut down
>>> 2011-12-29 17:14:59,897 - INFO  [regionserver7040:ZooKeeper@538] -
>>> Session: 0x1343f2669140013 closed
>>> 2011-12-29 17:14:59,898 - INFO  [regionserver7040:HRegionServer@686] -
>>> regionserver7040 exiting
>>> 2011-12-29 17:14:59,899 - INFO
>>> [regionserver7040-EventThread:ClientCnxn$EventThread@520] -
>>> EventThread shut down
>>> 2011-12-29 17:14:59,900 - INFO
>>> [Shutdownhook:regionserver7040:ShutdownHook$ShutdownHookThread@106] -
>>> Starting fs shutdown hook thread.
>>> 2011-12-29 17:14:59,900 - ERROR
>>> [Thread-15:DFSClient$LeaseChecker@1135] - Exception closing file
>>> /richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
>>> : org.apache.hadoop.ipc.RemoteException:
>>> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
>>> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
>>>        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1455)
>>>        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:649)
>>>        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
>>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1415)
>>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1411)
>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>        at javax.security.auth.Subject.doAs(Subject.java:396)
>>>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1409)
>>>
>>> org.apache.hadoop.ipc.RemoteException:
>>> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
>>> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423
>>>        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1455)
>>>        at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:649)
>>>        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
>>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1415)
>>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1411)
>>>        at java.security.AccessController.doPrivileged(Native Method)
>>>        at javax.security.auth.Subject.doAs(Subject.java:396)
>>>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1409)
>>>
>>>        at org.apache.hadoop.ipc.Client.call(Client.java:1104)
>>>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
>>>        at $Proxy5.addBlock(Unknown Source)
>>>        at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
>>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>>        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>        at $Proxy5.addBlock(Unknown Source)
>>>        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3185)
>>>        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3055)
>>>        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1900(DFSClient.java:2305)
>>>        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2500)
>>> 2011-12-29 17:14:59,913 - INFO
>>> [Shutdownhook:regionserver7040:ShutdownHook$ShutdownHookThread@112] -
>>> Shutdown hook finished.
>>> ------------------------

Mime
View raw message