Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CFFF29179 for ; Wed, 4 Jan 2012 00:15:16 +0000 (UTC) Received: (qmail 900 invoked by uid 500); 4 Jan 2012 00:15:15 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 814 invoked by uid 500); 4 Jan 2012 00:15:15 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 806 invoked by uid 99); 4 Jan 2012 00:15:14 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2012 00:15:14 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: unknown (nike.apache.org: error in processing during lookup of alok@urbanairship.com) Received: from [209.85.214.169] (HELO mail-tul01m020-f169.google.com) (209.85.214.169) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2012 00:15:08 +0000 Received: by obcwo8 with SMTP id wo8so17285900obc.14 for ; Tue, 03 Jan 2012 16:14:46 -0800 (PST) MIME-Version: 1.0 Received: by 10.50.182.199 with SMTP id eg7mr64957181igc.22.1325636086678; Tue, 03 Jan 2012 16:14:46 -0800 (PST) Received: by 10.42.136.132 with HTTP; Tue, 3 Jan 2012 16:14:46 -0800 (PST) In-Reply-To: References: Date: Tue, 3 Jan 2012 16:14:46 -0800 Message-ID: Subject: Re: NotReplicatedYetException when gracefully shutting down a region server. From: Alok Singh To: user@hbase.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org J-D, On the regionserver, I see these lines: ------------- 2011-12-28 18:56:37,821 - INFO [regionserver7040.cacheFlusher:Store@494] - Renaming flushed file at hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472= fb092a4470e9505c3a421a/.tmp/745069841898422651 to hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b= 472fb092a4470e9505c3a421a/user/8857592033950378228 2011-12-28 18:56:37,867 - INFO [regionserver7040.cacheFlusher:StoreFile$Reader@1027] - Loaded row bloom filter metadata for hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472= fb092a4470e9505c3a421a/user/8857592033950378228 2011-12-28 18:56:37,867 - INFO [regionserver7040.cacheFlusher:Store@504] - Added hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472= fb092a4470e9505c3a421a/user/8857592033950378228, entries=3D15990, sequenceid=3D1213107064, memsize=3D3.8m, filesize=3D200.1k 2011-12-28 19:22:57,674 - INFO [PRI IPC Server handler 9 on 7040:HRegionServer@2142] - Received close region: rich_push.user,4e3b951f2325b4695800679d,1317431456933.0fd5a7b472fb092a4470e= 9505c3a421a. 2011-12-28 19:22:57,702 - INFO [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:StoreFile$Writer@868] - Bloom added to HFile (hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b47= 2fb092a4470e9505c3a421a/.tmp/8759993386901964725): 32.0, 14/26 (54%) 2011-12-28 19:22:57,776 - INFO [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:Store@494] - Renaming flushed file at hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472= fb092a4470e9505c3a421a/.tmp/8759993386901964725 to hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b= 472fb092a4470e9505c3a421a/user/4476093570994008279 2011-12-28 19:22:57,893 - INFO [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:StoreFile$Reader@1027] - Loaded row bloom filter metadata for hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472= fb092a4470e9505c3a421a/user/4476093570994008279 2011-12-28 19:22:57,894 - INFO [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:Store@504] - Added hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user/0fd5a7b472= fb092a4470e9505c3a421a/user/4476093570994008279, entries=3D210, sequenceid=3D1213258636, memsize=3D50.8k, filesize=3D3.6k 2011-12-28 19:22:57,896 - INFO [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-2:HRegion@554] - Closed rich_push.user,4e3b951f2325b4695800679d,1317431456933.0fd5a7b472fb09= 2a4470e9505c3a421a. 2011-12-29 17:14:59,900 - ERROR [Thread-15:DFSClient$LeaseChecker@1135] - Exception closing file /richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6= 475368557525722423 : org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 ---------------------------------------- We waited overnight after moving the regions from the regionserver before shutting it down. That is why the timestamp between the RS_CLOSE_REGION and ERROR are off by 22 hours. But, looking on the namenode, I found some interesting log entries. Note the timestamp on these entries, they are over a week before the regionserver errors occurred. -------namenode-logs--------- rm=3Dnull 2011-12-20 03:47:41,208 - INFO [IPC Server handler 8 on 7080:FSNamesystem@125] - ugi=3Dhbaseregion ip=3D/10.129.1.230 cmd=3Dcreate src=3D/richpush/hbase/roo= t/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6475368557525722423 = dst=3Dnull perm=3Dhbaseregion:hadoop:rw-rw-rw- 2011-12-20 03:47:41,240 - INFO [IPC Server handler 60 on 7080:FSNamesystem@1646] - BLOCK* NameSystem.allocateBlock: /richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6= 475368557525722423. blk_4226819991054427111_66443 2011-12-20 03:48:10,096 - INFO [IPC Server handler 40 on 7080:FSNamesystem@1646] - BLOCK* NameSystem.allocateBlock: /richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tmp/6= 475368557525722423. blk_51125950084620321_66443 2011-12-20 03:48:38,808 - INFO [IPC Server handler 1 on 7080:Server$Handler@1423] - IPC Server handler 1 on 7080, call addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a42= 1a/.tmp/6475368557525722423, DFSClient_201969766, null) from 10.129.1.230:54052: error: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 2011-12-20 03:48:39,245 - INFO [IPC Server handler 22 on 7080:Server$Handler@1423] - IPC Server handler 22 on 7080, call addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a42= 1a/.tmp/6475368557525722423, DFSClient_201969766, null) from 10.129.1.230:54052: error: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 2011-12-20 03:48:40,055 - INFO [IPC Server handler 42 on 7080:Server$Handler@1423] - IPC Server handler 42 on 7080, call addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a42= 1a/.tmp/6475368557525722423, DFSClient_201969766, null) from 10.129.1.230:54052: error: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 2011-12-20 03:48:41,665 - INFO [IPC Server handler 63 on 7080:Server$Handler@1423] - IPC Server handler 63 on 7080, call addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a42= 1a/.tmp/6475368557525722423, DFSClient_201969766, null) from 10.129.1.230:54052: error: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 2011-12-20 03:48:44,875 - INFO [IPC Server handler 26 on 7080:Server$Handler@1423] - IPC Server handler 26 on 7080, call addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a42= 1a/.tmp/6475368557525722423, DFSClient_201969766, null) from 10.129.1.230:54052: error: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 2011-12-20 03:48:51,285 - INFO [IPC Server handler 11 on 7080:Server$Handler@1423] - IPC Server handler 11 on 7080, call addBlock(/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a42= 1a/.tmp/6475368557525722423, DFSClient_201969766, null) from 10.129.1.230:54052: error: org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e950= 5c3a421a/.tmp/6475368557525722423 2011-12-20 06:34:21,299 - INFO [IPC Server handler 10 on 7080:FSNamesystem@125] - ugi=3Dhbaseregion ip=3D/10.129.1.230 cmd=3DlistStatus src=3D/richpush/hbase= /root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/attribute dst=3Dnull = perm=3Dnull 2011-12-20 06:34:21,303 - INFO [IPC Server handler 19 on 7080:FSNamesystem@125] - ugi=3Dhbaseregion ip=3D/10.129.1.230 cmd=3DlistStatus src=3D/richpush/hbase= /root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/user dst=3Dnull perm= =3Dnull .... -----namenode-logs------------------ Thanks for helping me figure out this error. Alok On Tue, Jan 3, 2012 at 2:06 PM, Jean-Daniel Cryans wr= ote: > Since you did the region mover thing then all the region should be off > from that region server, so data loss is highly unlikely. > > My theory would be that a compaction was underway and got cancelled > when the region server was told to close the region, but the > DFSOutputStream thread hasn't been cleanup yet. Probably the file was > even already deleted and the data is still safe in the un-compacted > files. > > Ways to confirm this: > > =A0- Look in the region server log in the preceding lines what happens > to that region (named 0fd5a7b472fb092a4470e9505c3a421a) when it's > closing. Do you see a compaction being cancelled? > =A0- Look in the Namenode log for that file, is it being asked to delete > it or something before it sends a NotReplicatedYetException? > > Hope this helps, > > J-D > > On Thu, Dec 29, 2011 at 3:11 PM, Alok Singh wrote= : >> When attempting to gracefully shutdown a regionserver, I saw a couple >> of NotReplicatedYet exceptions in the logs (below). Can't find the >> file that is causing this exception in on the HDFS filesystem. Have we >> potentially lost the data, or is this exception benign? >> >> >> Alok >> >> hbase: 0.90.3 >> hadoop: 0.20.2-cdh3u0 >> >> Gracefull shutdown process: >> hbase(main):001:0> balance_switch false >> >> hbase org.jruby.Main current/bin/region_mover.rb unload >> >> After the region count for the regionserver is 0, we kill the >> regionserver process. >> >> ------------------Log------------------------- >> 2011-12-29 17:11:03,768 - INFO =A0[PRI IPC Server handler 6 on >> 7040:HRegionServer@2142] - Received close region: >> rich_push.user_device,ujoG9VuxSbKTLOQalFdJDA.c9QrCxywSJWvSfYhDGL1XA,1324= 711834779.01c7894eea30fcd6713b159ca1645da5. >> 2011-12-29 17:11:03,787 - WARN >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:FsPermission@220] - >> dfs.umask configuration key is deprecated. Convert to dfs.umaskmode, >> using octal or symbolic umask specifications. >> 2011-12-29 17:11:03,803 - WARN >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:FsPermission@220] - >> dfs.umask configuration key is deprecated. Convert to dfs.umaskmode, >> using octal or symbolic umask specifications. >> 2011-12-29 17:11:03,911 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:StoreFile$Writer@868] >> - Bloom added to HFile >> (hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_device= /01c7894eea30fcd6713b159ca1645da5/.tmp/6825595110476022990): >> 7.1k, 5225/5276 (99%) >> 2011-12-29 17:11:03,944 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:Store@494] - >> Renaming flushed file at >> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_device/= 01c7894eea30fcd6713b159ca1645da5/.tmp/6825595110476022990 >> to hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_devi= ce/01c7894eea30fcd6713b159ca1645da5/device/1580164973134104965 >> 2011-12-29 17:11:03,961 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:StoreFile$Reader@1027= ] >> - Loaded col bloom filter metadata for >> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_device/= 01c7894eea30fcd6713b159ca1645da5/device/1580164973134104965 >> 2011-12-29 17:11:03,961 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:Store@504] - Added >> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_device/= 01c7894eea30fcd6713b159ca1645da5/device/1580164973134104965, >> entries=3D5276, sequenceid=3D1219931822, memsize=3D2.0m, filesize=3D384.= 4k >> 2011-12-29 17:11:03,964 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-0:HRegion@554] - >> Closed rich_push.user_device,ujoG9VuxSbKTLOQalFdJDA.c9QrCxywSJWvSfYhDGL1= XA,1324711834779.01c7894eea30fcd6713b159ca1645da5. >> 2011-12-29 17:11:05,706 - INFO =A0[PRI IPC Server handler 7 on >> 7040:HRegionServer@2142] - Received close region: >> rich_push.user_udid,7sDqU74XQN6x1LWFPsmomQ.4e77ba299bf7e812c5002186,1325= 081217300.6f4cf99ff967f159fc68cd6c05001e9a. >> 2011-12-29 17:11:05,716 - WARN >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:FsPermission@220] - >> dfs.umask configuration key is deprecated. Convert to dfs.umaskmode, >> using octal or symbolic umask specifications. >> 2011-12-29 17:11:05,735 - WARN >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:FsPermission@220] - >> dfs.umask configuration key is deprecated. Convert to dfs.umaskmode, >> using octal or symbolic umask specifications. >> 2011-12-29 17:11:05,808 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:StoreFile$Writer@868] >> - Bloom added to HFile >> (hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid/6= f4cf99ff967f159fc68cd6c05001e9a/.tmp/1207401926740467097): >> 6.5k, 4727/4737 (100%) >> 2011-12-29 17:11:05,834 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:Store@494] - >> Renaming flushed file at >> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid/6f= 4cf99ff967f159fc68cd6c05001e9a/.tmp/1207401926740467097 >> to hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid= /6f4cf99ff967f159fc68cd6c05001e9a/udid/8492436804961336492 >> 2011-12-29 17:11:05,852 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:StoreFile$Reader@1027= ] >> - Loaded col bloom filter metadata for >> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid/6f= 4cf99ff967f159fc68cd6c05001e9a/udid/8492436804961336492 >> 2011-12-29 17:11:05,852 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:Store@504] - Added >> hdfs://richpush-master-0:7080/richpush/hbase/root/rich_push.user_udid/6f= 4cf99ff967f159fc68cd6c05001e9a/udid/8492436804961336492, >> entries=3D4737, sequenceid=3D1219931829, memsize=3D1.4m, filesize=3D246.= 5k >> 2011-12-29 17:11:05,856 - INFO >> [RS_CLOSE_REGION-10.129.1.230,7040,1323990853621-1:HRegion@554] - >> Closed rich_push.user_udid,7sDqU74XQN6x1LWFPsmomQ.4e77ba299bf7e812c50021= 86,1325081217300.6f4cf99ff967f159fc68cd6c05001e9a. >> 2011-12-29 17:14:58,942 - INFO >> [Shutdownhook:regionserver7040:ShutdownHook$ShutdownHookThread@100] - >> Shutdown hook starting; hbase.shutdown.hook=3Dtrue; >> fsShutdownHook=3DThread[Thread-15,5,main] >> 2011-12-29 17:14:58,942 - INFO >> [Shutdownhook:regionserver7040:HRegionServer@1342] - STOPPED: Shutdown >> hook >> 2011-12-29 17:14:59,476 - INFO =A0[regionserver7040:HBaseServer@1234] - >> Stopping server on 7040 >> 2011-12-29 17:14:59,476 - INFO =A0[PRI IPC Server handler 2 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 2 on 7040: >> exiting >> 2011-12-29 17:14:59,476 - INFO =A0[IPC Server handler 9 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 9 on 7040: exiting >> 2011-12-29 17:14:59,476 - INFO =A0[PRI IPC Server handler 4 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 4 on 7040: >> exiting >> 2011-12-29 17:14:59,476 - INFO =A0[IPC Server handler 4 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 4 on 7040: exiting >> 2011-12-29 17:14:59,477 - INFO =A0[IPC Server handler 5 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 5 on 7040: exiting >> 2011-12-29 17:14:59,477 - INFO =A0[IPC Server handler 6 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 6 on 7040: exiting >> 2011-12-29 17:14:59,477 - INFO =A0[IPC Server handler 7 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 7 on 7040: exiting >> 2011-12-29 17:14:59,477 - INFO =A0[IPC Server handler 8 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 8 on 7040: exiting >> 2011-12-29 17:14:59,477 - INFO =A0[PRI IPC Server handler 0 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 0 on 7040: >> exiting >> 2011-12-29 17:14:59,478 - INFO =A0[PRI IPC Server handler 1 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 1 on 7040: >> exiting >> 2011-12-29 17:14:59,478 - INFO =A0[PRI IPC Server handler 3 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 3 on 7040: >> exiting >> 2011-12-29 17:14:59,478 - INFO =A0[PRI IPC Server handler 5 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 5 on 7040: >> exiting >> 2011-12-29 17:14:59,478 - INFO =A0[PRI IPC Server handler 6 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 6 on 7040: >> exiting >> 2011-12-29 17:14:59,478 - INFO =A0[PRI IPC Server handler 7 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 7 on 7040: >> exiting >> 2011-12-29 17:14:59,478 - INFO =A0[PRI IPC Server handler 9 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 9 on 7040: >> exiting >> 2011-12-29 17:14:59,476 - INFO =A0[IPC Server handler 0 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 0 on 7040: exiting >> 2011-12-29 17:14:59,480 - INFO =A0[IPC Server >> Responder:HBaseServer$Responder@649] - Stopping IPC Server Responder >> 2011-12-29 17:14:59,479 - INFO =A0[IPC Server listener on >> 7040:HBaseServer$Listener@450] - Stopping IPC Server listener on 7040 >> 2011-12-29 17:14:59,477 - INFO =A0[regionserver7040:HRegionServer@636] - >> Stopping infoServer >> 2011-12-29 17:14:59,476 - INFO =A0[PRI IPC Server handler 8 on >> 7040:HBaseServer$Handler@1104] - PRI IPC Server handler 8 on 7040: >> exiting >> 2011-12-29 17:14:59,476 - INFO =A0[IPC Server handler 3 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 3 on 7040: exiting >> 2011-12-29 17:14:59,476 - INFO =A0[IPC Server handler 2 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 2 on 7040: exiting >> 2011-12-29 17:14:59,476 - INFO =A0[IPC Server handler 1 on >> 7040:HBaseServer$Handler@1104] - IPC Server handler 1 on 7040: exiting >> 2011-12-29 17:14:59,481 - INFO =A0[regionserver7040:Slf4jLog@67] - >> Stopped SelectChannelConnector@0.0.0.0:7041 >> 2011-12-29 17:14:59,517 - INFO >> [regionserver7040.logRoller:LogRoller@114] - LogRoller exiting. >> 2011-12-29 17:14:59,517 - INFO >> [regionserver7040.cacheFlusher:MemStoreFlusher@266] - >> regionserver7040.cacheFlusher exiting >> 2011-12-29 17:14:59,517 - INFO >> [regionserver7040.logSyncer:HLog$LogSyncer@966] - >> regionserver7040.logSyncer exiting >> 2011-12-29 17:14:59,518 - INFO >> [regionserver7040.majorCompactionChecker:Chore@79] - >> regionserver7040.majorCompactionChecker exiting >> 2011-12-29 17:14:59,518 - INFO >> [regionserver7040.compactor:CompactSplitThread@113] - >> regionserver7040.compactor exiting >> 2011-12-29 17:14:59,891 - INFO =A0[regionserver7040:HRegionServer@668] - >> stopping server at: 10.129.1.230,7040,1323990853621 >> 2011-12-29 17:14:59,891 - INFO =A0[regionserver7040:Leases@124] - >> regionserver7040 closing leases >> 2011-12-29 17:14:59,891 - INFO =A0[regionserver7040:Leases@131] - >> regionserver7040 closed leases >> 2011-12-29 17:14:59,891 - INFO >> [regionserver7040:HConnectionManager$HConnectionImplementation@1067] - >> Closed zookeeper sessionid=3D0x4343f266cbc0012 >> 2011-12-29 17:14:59,894 - INFO =A0[regionserver7040:ZooKeeper@538] - >> Session: 0x4343f266cbc0012 closed >> 2011-12-29 17:14:59,894 - INFO >> [main-EventThread:ClientCnxn$EventThread@520] - EventThread shut down >> 2011-12-29 17:14:59,897 - INFO =A0[regionserver7040:ZooKeeper@538] - >> Session: 0x1343f2669140013 closed >> 2011-12-29 17:14:59,898 - INFO =A0[regionserver7040:HRegionServer@686] - >> regionserver7040 exiting >> 2011-12-29 17:14:59,899 - INFO >> [regionserver7040-EventThread:ClientCnxn$EventThread@520] - >> EventThread shut down >> 2011-12-29 17:14:59,900 - INFO >> [Shutdownhook:regionserver7040:ShutdownHook$ShutdownHookThread@106] - >> Starting fs shutdown hook thread. >> 2011-12-29 17:14:59,900 - ERROR >> [Thread-15:DFSClient$LeaseChecker@1135] - Exception closing file >> /richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e9505c3a421a/.tm= p/6475368557525722423 >> : org.apache.hadoop.ipc.RemoteException: >> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not >> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e= 9505c3a421a/.tmp/6475368557525722423 >> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ge= tAdditionalBlock(FSNamesystem.java:1455) >> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlo= ck(NameNode.java:649) >> =A0 =A0 =A0 =A0at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown So= urce) >> =A0 =A0 =A0 =A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delega= tingMethodAccessorImpl.java:25) >> =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Method.java:597) >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java= :1415) >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java= :1411) >> =A0 =A0 =A0 =A0at java.security.AccessController.doPrivileged(Native Met= hod) >> =A0 =A0 =A0 =A0at javax.security.auth.Subject.doAs(Subject.java:396) >> =A0 =A0 =A0 =A0at org.apache.hadoop.security.UserGroupInformation.doAs(U= serGroupInformation.java:1115) >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1= 409) >> >> org.apache.hadoop.ipc.RemoteException: >> org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not >> replicated yet:/richpush/hbase/root/rich_push.user/0fd5a7b472fb092a4470e= 9505c3a421a/.tmp/6475368557525722423 >> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ge= tAdditionalBlock(FSNamesystem.java:1455) >> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlo= ck(NameNode.java:649) >> =A0 =A0 =A0 =A0at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown So= urce) >> =A0 =A0 =A0 =A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delega= tingMethodAccessorImpl.java:25) >> =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Method.java:597) >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java= :1415) >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java= :1411) >> =A0 =A0 =A0 =A0at java.security.AccessController.doPrivileged(Native Met= hod) >> =A0 =A0 =A0 =A0at javax.security.auth.Subject.doAs(Subject.java:396) >> =A0 =A0 =A0 =A0at org.apache.hadoop.security.UserGroupInformation.doAs(U= serGroupInformation.java:1115) >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1= 409) >> >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.Client.call(Client.java:1104) >> =A0 =A0 =A0 =A0at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) >> =A0 =A0 =A0 =A0at $Proxy5.addBlock(Unknown Source) >> =A0 =A0 =A0 =A0at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown S= ource) >> =A0 =A0 =A0 =A0at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delega= tingMethodAccessorImpl.java:25) >> =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Method.java:597) >> =A0 =A0 =A0 =A0at org.apache.hadoop.io.retry.RetryInvocationHandler.invo= keMethod(RetryInvocationHandler.java:82) >> =A0 =A0 =A0 =A0at org.apache.hadoop.io.retry.RetryInvocationHandler.invo= ke(RetryInvocationHandler.java:59) >> =A0 =A0 =A0 =A0at $Proxy5.addBlock(Unknown Source) >> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locat= eFollowingBlock(DFSClient.java:3185) >> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextB= lockOutputStream(DFSClient.java:3055) >> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.acces= s$1900(DFSClient.java:2305) >> =A0 =A0 =A0 =A0at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataS= treamer.run(DFSClient.java:2500) >> 2011-12-29 17:14:59,913 - INFO >> [Shutdownhook:regionserver7040:ShutdownHook$ShutdownHookThread@112] - >> Shutdown hook finished. >> ------------------------