Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 431CB11DF for ; Tue, 26 Apr 2011 21:55:43 +0000 (UTC) Received: (qmail 29259 invoked by uid 500); 26 Apr 2011 21:55:43 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 29233 invoked by uid 500); 26 Apr 2011 21:55:43 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 29223 invoked by uid 99); 26 Apr 2011 21:55:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Apr 2011 21:55:43 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.116] (HELO hel.zones.apache.org) (140.211.11.116) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Apr 2011 21:55:41 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [140.211.11.116]) by hel.zones.apache.org (Postfix) with ESMTP id 57630B5B0C for ; Tue, 26 Apr 2011 21:55:03 +0000 (UTC) Date: Tue, 26 Apr 2011 21:55:03 +0000 (UTC) From: "Prakash Khemani (JIRA)" To: issues@hbase.apache.org Message-ID: <1782221730.4084.1303854903354.JavaMail.tomcat@hel.zones.apache.org> Subject: [jira] [Created] (HBASE-3824) region server timed out during open region MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 region server timed out during open region ------------------------------------------ Key: HBASE-3824 URL: https://issues.apache.org/jira/browse/HBASE-3824 Project: HBase Issue Type: Bug Reporter: Prakash Khemani When replaying a large log file, mestore flushes can happen. But there is no Progressible report being sent during memstore flushes. That can lead to master timing out the region server during region open. === Another related issue and Jonathan's response > So if a region server that is handed a region for opening and has done part of > the work ... it has created some HFiles (because the logs were so huge that > the mestore got flushed while the logs were being replayed) ... and then it is > asked to give up because the master thought the region server was taking > too long to open the region. > > When the region server gives up on the region then will it make sure that it > removes all the HFiles it had created for that region? Will need to check the code, but would it matter? One issue is whether it cleans up after itself (I'm guessing not). Another issue is whether the replay is idempotent (duplicate KVs across files shouldn't matter in most cases). === 2011-04-25 09:11:36,844 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_RS_OPEN_REGION java.lang.NullPointerException at org.apache.hadoop.hbase.util.Writables.getWritable(Writables.java:75) at org.apache.hadoop.hbase.executor.RegionTransitionData.fromBytes(RegionTransitionData.java:198) at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNode(ZKAssign.java:672) at org.apache.hadoop.hbase.zookeeper.ZKAssign.transitionNodeOpened(ZKAssign.java:621) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:168) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) byte [] existingBytes = ZKUtil.getDataNoWatch(zkw, node, stat); RegionTransitionData existingData = RegionTransitionData.fromBytes(existingBytes); existingBytes can be null. have to return -1 if null. === master logs 2011-04-25 05:24:03,250 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Creating writer path=hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/recovered.edits/0000000057528037047 region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:09:19,246 INFO org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Closed path hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/recovered.edits/0000000057528037047 (wrote 4342690 edits in 46904ms) 2011-04-25 09:09:26,134 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x32f7bb74e8a0000 Creating (or updating) unassigned node for e7a478b4bd164525052f1dedb832de0a with OFFLINE state 2011-04-25 09:09:26,136 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. so generated a random one; hri=realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a., src=, dest=pumahbase107.snc5.facebook.com,60020,1303450731227; 70 (online=70, exclude=null) available servers 2011-04-25 09:09:26,136 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. to pumahbase107.snc5.facebook.com,60020,1303450731227 2011-04-25 09:09:26,139 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase107.snc5.facebook.com,60020,1303450731227, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:09:44,045 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase107.snc5.facebook.com,60020,1303450731227, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:09:59,050 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase107.snc5.facebook.com,60020,1303450731227, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:10:14,054 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase107.snc5.facebook.com,60020,1303450731227, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:10:29,055 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase107.snc5.facebook.com,60020,1303450731227, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:10:44,060 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase107.snc5.facebook.com,60020,1303450731227, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:10:59,062 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase107.snc5.facebook.com,60020,1303450731227, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:10:59,388 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase107.snc5.facebook.com,60020,1303450731227, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:11:33,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. state=OPENING, ts=1303747859386 2011-04-25 09:11:33,411 INFO org.apache.hadoop.hbase.master.AssignmentManager: Region has been OPENING for too long, reassigning region=realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. 2011-04-25 09:11:33,412 INFO org.apache.hadoop.hbase.master.AssignmentManager: Successfully transitioned region=realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. into OFFLINE and forcing a new assignment 2011-04-25 09:11:33,412 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. so generated a random one; hri=realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a., src=, dest=pumahbase150.snc5.facebook.com,60020,1303450731207; 70 (online=70, exclude=null) available servers 2011-04-25 09:11:33,413 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. to pumahbase150.snc5.facebook.com,60020,1303450731207 2011-04-25 09:11:33,414 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=pumahbase162.snc5.facebook.com:60000, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:11:33,416 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase150.snc5.facebook.com,60020,1303450731207, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:11:33,825 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=pumahbase150.snc5.facebook.com,60020,1303450731207, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:11:36,804 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=pumahbase150.snc5.facebook.com,60020,1303450731207, region=e7a478b4bd164525052f1dedb832de0a 2011-04-25 09:11:36,804 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for e7a478b4bd164525052f1dedb832de0a; deleting unassigned node 2011-04-25 09:11:36,804 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x32f7bb74e8a0000 Deleting existing unassigned node for e7a478b4bd164525052f1dedb832de0a that is in expected state RS_ZK_REGION_OPENED 2011-04-25 09:11:36,805 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x32f7bb74e8a0000 Successfully deleted unassigned node for region e7a478b4bd164525052f1dedb832de0a in expected state RS_ZK_REGION_OPENED 2011-04-25 09:11:36,805 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Opened region realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. on pumahbase150.snc5.facebook.com,60020,1303450731207 === region server log 2011-04-25 09:09:26,136 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. 2011-04-25 09:09:26,136 DEBUG org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Processing open of realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. 2011-04-25 09:09:26,136 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Attempting to transition node e7a478b4bd164525052f1dedb832de0a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2011-04-25 09:09:26,138 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Successfully transitioned node e7a478b4bd164525052f1dedb832de0a from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2011-04-25 09:09:26,139 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: REGION => {NAME => 'realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a.', STARTKEY => 'afffffbe', ENDKEY => 'b040ebdbf81a3750f0eaa71842ff09dagr.insomnia li 294576344998', ENCODED => e7a478b4bd164525052f1dedb832de0a, TABLE => {{NAME => 'realtime_domain_imps_urls', FAMILIES => [{NAME => 'COUNTERS_0', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'LZO', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => '16384', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'COUNTERS_3600', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'LZO', VERSIONS => '1', TTL => '172800', BLOCKSIZE => '16384', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'COUNTERS_86400', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'LZO', VERSIONS => '1', TTL => '2592000', BLOCKSIZE => '16384', IN_MEMORY => 'false', BLOCKCACHE => 't rue'}]}} 2011-04-25 09:09:26,140 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. 2011-04-25 09:09:26,140 INFO org.apache.hadoop.hbase.regionserver.logger.HRegionLogger: HRegionLogger for region realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a., category nectar_titan_hbase_updates 2011-04-25 09:09:26,190 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_0/87cf11f68701404b862d631ba3d20213, isReference=false, isBulkLoadResult=false, seqid=58137870275, majorCompaction=false 2011-04-25 09:09:28,572 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_0/c58d9a33653a4e5e966c426b33a57598, isReference=false, isBulkLoadResult=false, seqid=57847795298, majorCompaction=true 2011-04-25 09:09:28,796 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_3600/1bc513e915b741d59d4f113c273e67d4, isReference=false, isBulkLoadResult=false, seqid=58137870275, majorCompaction=false 2011-04-25 09:09:28,826 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_3600/5a50b17f3ad045368229680414987ae7, isReference=false, isBulkLoadResult=false, seqid=57847795298, majorCompaction=true 2011-04-25 09:09:28,850 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_86400/23648dce8d4243c5b804c0e766f5a235, isReference=false, isBulkLoadResult=false, seqid=58137870275, majorCompaction=false 2011-04-25 09:09:29,016 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_86400/d6820d86715049c087fa5c3f4355f2fc, isReference=false, isBulkLoadResult=false, seqid=57847795298, majorCompaction=true 2011-04-25 09:09:29,017 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/recovered.edits/0000000057528037047; minSequenceid=58137870275 2011-04-25 09:09:44,041 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Attempting to transition node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:09:44,044 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Successfully transitioned node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:09:59,046 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Attempting to transition node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:09:59,050 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Successfully transitioned node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:10:14,050 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Attempting to transition node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:10:29,055 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Successfully transitioned node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:10:44,056 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Attempting to transition node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:10:44,059 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Successfully transitioned node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:10:59,057 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Attempting to transition node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:10:59,062 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Successfully transitioned node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:10:59,154 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a., current region memstore size 18.9m; wal is null, using passed sequenceid=58140219688 2011-04-25 09:10:59,277 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=58140219688, memsize=12.4m, into tmp file hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/.tmp/a1df4f1ac37a4c7fbf4067f3bcff49a5 2011-04-25 09:10:59,312 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=58140219688, memsize=3.2m, into tmp file hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/.tmp/eefd1540ca644f06b251201894ef5dd8 2011-04-25 09:10:59,357 INFO org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=58140219688, memsize=3.2m, into tmp file hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/.tmp/83c87fcdbf7147ceac873a1e97e2990f 2011-04-25 09:10:59,357 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/.tmp/a1df4f1ac37a4c7fbf4067f3bcff49a5 to hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_0/a1df4f1ac37a4c7fbf4067f3bcff49a5 2011-04-25 09:10:59,372 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_0/a1df4f1ac37a4c7fbf4067f3bcff49a5, entries=45626, sequenceid=58140219688, filesize=549.1k 2011-04-25 09:10:59,372 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/.tmp/eefd1540ca644f06b251201894ef5dd8 to hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_3600/eefd1540ca644f06b251201894ef5dd8 2011-04-25 09:10:59,375 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_3600/eefd1540ca644f06b251201894ef5dd8, entries=11411, sequenceid=58140219688, filesize=167.9k 2011-04-25 09:10:59,375 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming flushed file at hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/.tmp/83c87fcdbf7147ceac873a1e97e2990f to hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_86400/83c87fcdbf7147ceac873a1e97e2990f 2011-04-25 09:10:59,378 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/COUNTERS_86400/83c87fcdbf7147ceac873a1e97e2990f, entries=11404, sequenceid=58140219688, filesize=161.5k 2011-04-25 09:10:59,378 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~18.9m for region realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. in 224ms, sequenceid=58140219688, compaction requested=true; wal=null 2011-04-25 09:10:59,385 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Deleted recovered.edits file=hdfs://pumahbase002-snc5-dfs.data.facebook.com:9000/PUMAHBASE002-SNC5-HBASE/realtime_domain_imps_urls/e7a478b4bd164525052f1dedb832de0a/recovered.edits/0000000057528037047 2011-04-25 09:10:59,385 INFO org.apache.hadoop.hbase.regionserver.HRegion: Onlined realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a.; next sequenceid=58140219689 2011-04-25 09:10:59,385 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Attempting to transition node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:10:59,388 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Successfully transitioned node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2011-04-25 09:11:36,843 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Updated row realtime_domain_imps_urls,afffffbe,1295556905482.e7a478b4bd164525052f1dedb832de0a. in region .META.,,1 with server=pumahbase107.snc5.facebook.com:60020, startcode=1303450731227 2011-04-25 09:11:36,843 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x22f7bb74e8e000a Attempting to transition node e7a478b4bd164525052f1dedb832de0a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED java.lang.NullPointerException === -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira