Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1B8951878F for ; Thu, 12 Nov 2015 17:56:39 +0000 (UTC) Received: (qmail 49389 invoked by uid 500); 12 Nov 2015 17:56:37 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 49326 invoked by uid 500); 12 Nov 2015 17:56:37 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 49313 invoked by uid 99); 12 Nov 2015 17:56:36 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Nov 2015 17:56:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 46B6F1A20B6 for ; Thu, 12 Nov 2015 17:56:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.151 X-Spam-Level: *** X-Spam-Status: No, score=3.151 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, KAM_LOTSOFHASH=0.25, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id XaOJKeVdfc7J for ; Thu, 12 Nov 2015 17:56:12 +0000 (UTC) Received: from mail-yk0-f193.google.com (mail-yk0-f193.google.com [209.85.160.193]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 0476620489 for ; Thu, 12 Nov 2015 17:56:12 +0000 (UTC) Received: by ykba77 with SMTP id a77so9871298ykb.2 for ; Thu, 12 Nov 2015 09:56:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=BAJ1mZZnJ8IOQbqNEexGpoa7tlWkxZOBM60xOA8IOgQ=; b=jqO1yyUMOKc6SXVdEquR4pPeoWrdpT3VGHTwvqqSomaA36/J5tvmEyuNKfLe9x6eoe 96WsFRIu0AI9KzILes8NkiC+cozoBUH0iN1Exv8dvTGQk9AeqdSfliG9I5HRcS5sNjsm C1ICTitznZncW0ptoVrknjfjspkse6UDsXRKlp2HwZ/PFJNxmxPOgEH2bAPDGtNPNAN1 VKf9Ypb5tSKZxSIh102ei4y8ur+kO/g7FQRF6eA11g1zAzIFm7YAoUvukXiYOB4M7Udr q3aLSOMu080loildX92PPAso1Q4Ds3qeaOPVzs389dP78SMFcT+Ohoe9Fitby1teix1D 834A== MIME-Version: 1.0 X-Received: by 10.129.114.213 with SMTP id n204mr10441750ywc.72.1447350965588; Thu, 12 Nov 2015 09:56:05 -0800 (PST) Received: by 10.103.79.141 with HTTP; Thu, 12 Nov 2015 09:56:05 -0800 (PST) Reply-To: work2much@gmail.com In-Reply-To: References: Date: Thu, 12 Nov 2015 09:56:05 -0800 Message-ID: Subject: Re: hbase regions behaving badly From: Abraham Tom To: Ted Yu Cc: "user@hbase.apache.org" Content-Type: multipart/alternative; boundary=001a11473af62b72c505245ba83f --001a11473af62b72c505245ba83f Content-Type: text/plain; charset=UTF-8 HDFS health via fsck shows healthy hbck shows no inconsistencies once all the regions finish transitioning What I ended up doing is setting hbase.hregion.max.filesize to 4 GB from he default CDH setting of 10GB. The old cluster had it at 10GB when I snapshot it and exported it. I did the normal flush compact after restarting. Then I let it load some data which actually forced split files again. I stopped loading and I let hbase do its transfers and splitting Once the transfers were done, enabled blockcache to be true on all tables and raised the hbase.hregion.max.filesize to 4.8 gb Now it seems to be stable, at least its not frantically splitting and resplitting regions since my new limit is higher than my old limit The primary reason for the change is that S3 only allows for files less than 5GB in size. I am guessing that auto region splitting is literally the size you specify and if you don't give it some growth room for random updates it will cause files to be split which could snowball out to larger overhead activity On Thu, Nov 12, 2015 at 6:53 AM, Ted Yu wrote: > bq. 2015-11-11 12:05:35,179 DEBUG > org.apache.hadoop.hdfs.BlockReaderFactory: Closed potentially stale remote > peer > > Have you checked hdfs health ? > > Cheers > > On Wed, Nov 11, 2015 at 10:10 AM, Abraham Tom wrote: > >> Thanks >> >> I also restarted in debug mode >> and found that hbase is renewing my lease to other nodes rather quickly >> logs here >> http://pastebin.com/4EgsXCDd >> >> >> On Wed, Nov 11, 2015 at 6:20 AM, Ted Yu wrote: >> >>> Please note that IncreasingToUpperBoundRegionSplitPolicy was in effect. >>> >>> See http://hbase.apache.org/book.html#_custom_split_policies for >>> related information (there is link to >>> http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.html >>> ). >>> >>> In the future, please consider using pastebin etc. for server log so >>> that the message is concise. >>> >>> Cheers >>> >>> On Tue, Nov 10, 2015 at 10:17 PM, Abraham Tom >>> wrote: >>> >>>> my hbase-site.xml snippet >>>> >>>> >>>> hbase.mob.sweep.tool.compaction.memstore.flush.size134217728hbase-default.xml >>>> >>>> hbase.hregion.memstore.flush.size134217728hbase-default.xml >>>> >>>> hbase.hregion.memstore.block.multiplier4hbase-default.xml >>>> >>>> hbase.mob.sweep.tool.compaction.memstore.flush.size134217728hbase-default.xml >>>> >>>> dfs.blockreport.split.threshold1000000hdfs-default.xml >>>> >>>> hbase.regionserver.regionSplitLimit1000hbase-default.xml >>>> >>>> hbase.splitlog.manager.timeout120000hbase-site.xml >>>> >>>> hbase.regionserver.region.split.policyorg.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicyhbase-default.xml >>>> >>>> mapreduce.job.split.metainfo.maxsize10000000mapred-default.xml >>>> >>>> >>>> >>>> >>>> >>>> and some master logs >>>> tail -100 >>>> hbase-cmf-hbase-MASTER-hadoop4-private.wdc01.infra.ripple.com.log.out >>>> 2015-11-10 22:31:16,205 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {5097e5a15891c375eca53b992584c52e state=PENDING_OPEN, ts=1447216273284, >>>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>>> {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>>> 2015-11-10 22:31:16,206 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> d3b39413ee1bc88a100cd96564c4c420 from >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>>> 2015-11-10 22:31:16,206 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {cff8a38c23c3987fbbebec615f45f4ac state=OPENING, ts=1447216276185, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>>> {cff8a38c23c3987fbbebec615f45f4ac state=OPEN, ts=1447216276206, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>>> 2015-11-10 22:31:16,206 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> bc842e97c8ff7b0f3498b38510a9c1c1 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,206 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {c73147aef08804451c26c1f8099008f2 state=PENDING_OPEN, ts=1447216273284, >>>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>>> {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>>> 2015-11-10 22:31:16,207 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> cff8a38c23c3987fbbebec615f45f4ac from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,208 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {8a671fb436fb0d186b37211d847aa897 state=OPENING, ts=1447216276188, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {8a671fb436fb0d186b37211d847aa897 state=OPEN, ts=1447216276208, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,209 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 8a671fb436fb0d186b37211d847aa897 from >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>>> 2015-11-10 22:31:16,209 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=PENDING_OPEN, ts=1447216273276, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,210 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {1258fdeaf4f00989a8282e5f9c215d7e state=OPENING, ts=1447216276193, server= >>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} to >>>> {1258fdeaf4f00989a8282e5f9c215d7e state=OPEN, ts=1447216276210, server= >>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} >>>> 2015-11-10 22:31:16,211 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 1258fdeaf4f00989a8282e5f9c215d7e from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,213 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {02666182c37df5043acdcaca31ceb095 state=OPENING, ts=1447216276184, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>>> {02666182c37df5043acdcaca31ceb095 state=OPEN, ts=1447216276213, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>>> 2015-11-10 22:31:16,214 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=PENDING_OPEN, ts=1447216273284, >>>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>>> 2015-11-10 22:31:16,215 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 02666182c37df5043acdcaca31ceb095 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,216 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {4c694bcbe9679747e738a630aec33c3c state=OPENING, ts=1447216276200, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {4c694bcbe9679747e738a630aec33c3c state=OPEN, ts=1447216276216, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,217 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 4c694bcbe9679747e738a630aec33c3c from >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803 >>>> 2015-11-10 22:31:16,217 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {dc3169f842a4d19014f040c887b9acff state=PENDING_OPEN, ts=1447216273280, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,219 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {8320a058823353a19f69a10da6827953 state=OPENING, ts=1447216276198, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {8320a058823353a19f69a10da6827953 state=OPEN, ts=1447216276219, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,219 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 8320a058823353a19f69a10da6827953 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,220 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {edfd9220fe6c2a37fe7521101d490941 state=PENDING_OPEN, ts=1447216273142, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,220 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {1937fd49e94d42fe072a29ff7af885bb state=OPENING, ts=1447216276205, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>>> {1937fd49e94d42fe072a29ff7af885bb state=OPEN, ts=1447216276220, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>>> 2015-11-10 22:31:16,220 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {e4b861ca490c05fa4276e73e629b471a state=OPENING, ts=1447216276204, server= >>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} to >>>> {e4b861ca490c05fa4276e73e629b471a state=OPEN, ts=1447216276220, server= >>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} >>>> 2015-11-10 22:31:16,221 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 1937fd49e94d42fe072a29ff7af885bb from >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803 >>>> 2015-11-10 22:31:16,221 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=PENDING_OPEN, ts=1447216273274, >>>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>>> 2015-11-10 22:31:16,221 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> e4b861ca490c05fa4276e73e629b471a from >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>>> 2015-11-10 22:31:16,222 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {2c7572eeff0a72eceec10e0301d344a7 state=OPENING, ts=1447216276205, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>>> {2c7572eeff0a72eceec10e0301d344a7 state=OPEN, ts=1447216276222, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>>> 2015-11-10 22:31:16,222 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 2c7572eeff0a72eceec10e0301d344a7 from >>>> hadoop3-private.wdc01.infra.ripple.com,60020,1447208454196 >>>> 2015-11-10 22:31:16,223 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {cb7e639dbb58b1e47c1da538165c503c state=PENDING_OPEN, ts=1447216273288, >>>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>>> {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>>> 2015-11-10 22:31:16,225 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>>> {5097e5a15891c375eca53b992584c52e state=OPEN, ts=1447216276225, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>>> 2015-11-10 22:31:16,225 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>>> {c73147aef08804451c26c1f8099008f2 state=OPEN, ts=1447216276225, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>>> 2015-11-10 22:31:16,226 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {abddf225d895bd30e11205390fedc03e state=OPENING, ts=1447216276205, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>>> {abddf225d895bd30e11205390fedc03e state=OPEN, ts=1447216276226, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>>> 2015-11-10 22:31:16,226 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 5097e5a15891c375eca53b992584c52e from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,226 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> c73147aef08804451c26c1f8099008f2 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,227 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> abddf225d895bd30e11205390fedc03e from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,227 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {a5e6796ae343481b92b0180d817c1e17 state=PENDING_OPEN, ts=1447216273288, >>>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>>> {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>>> 2015-11-10 22:31:16,229 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPEN, ts=1447216276229, server= >>>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>>> 2015-11-10 22:31:16,230 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPEN, ts=1447216276230, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,230 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> dae9fb9b19a8d2076fd4be2b95aefe45 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,230 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> e920fab780cdd6ba183bc50a8b5d8c72 from >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>>> 2015-11-10 22:31:16,230 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {2c04351380cc266df609ab53ef0e56ad state=PENDING_OPEN, ts=1447216273276, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,236 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {edfd9220fe6c2a37fe7521101d490941 state=OPEN, ts=1447216276236, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,236 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPEN, ts=1447216276236, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>>> 2015-11-10 22:31:16,237 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {dc3169f842a4d19014f040c887b9acff state=OPEN, ts=1447216276237, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,240 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> edfd9220fe6c2a37fe7521101d490941 from >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803 >>>> 2015-11-10 22:31:16,240 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=PENDING_OPEN, ts=1447216273280, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,241 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {c2a9f77791512281e4491e0797e824e8 state=PENDING_OPEN, ts=1447216273277, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,241 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> dc3169f842a4d19014f040c887b9acff from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,241 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 9ecf375c160ffe4616b8c4fb45ea8fba from >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>>> 2015-11-10 22:31:16,244 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>>> {cb7e639dbb58b1e47c1da538165c503c state=OPEN, ts=1447216276244, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>>> 2015-11-10 22:31:16,245 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> cb7e639dbb58b1e47c1da538165c503c from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,246 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>>> {a5e6796ae343481b92b0180d817c1e17 state=OPEN, ts=1447216276246, server= >>>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>>> 2015-11-10 22:31:16,246 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> a5e6796ae343481b92b0180d817c1e17 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,252 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {2c04351380cc266df609ab53ef0e56ad state=OPEN, ts=1447216276252, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,253 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 2c04351380cc266df609ab53ef0e56ad from >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>>> 2015-11-10 22:31:16,253 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {15701b8131ca505f6cce148d49cfbeee state=PENDING_OPEN, ts=1447216273280, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,257 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPEN, ts=1447216276257, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,257 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {c2a9f77791512281e4491e0797e824e8 state=OPEN, ts=1447216276257, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,258 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> a4f70c6c01f31f6d3d65e0cf55200c80 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,258 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> c2a9f77791512281e4491e0797e824e8 from >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>>> 2015-11-10 22:31:16,258 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {7ce823a4e0f56bf8c854175409bb17b6 state=PENDING_OPEN, ts=1447216273280, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,258 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {1fbbd37bdff125be32d352c98dffd2b1 state=PENDING_OPEN, ts=1447216273280, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,280 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPEN, ts=1447216276280, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,280 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPEN, ts=1447216276280, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,281 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 1fbbd37bdff125be32d352c98dffd2b1 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,281 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 7ce823a4e0f56bf8c854175409bb17b6 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,281 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {cc52e0ba23708df10d696d0e33d9d137 state=PENDING_OPEN, ts=1447216273280, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,281 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {21b24baeaf26e12d02f79f243076850a state=PENDING_OPEN, ts=1447216273280, >>>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,306 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {15701b8131ca505f6cce148d49cfbeee state=OPEN, ts=1447216276306, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,307 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 15701b8131ca505f6cce148d49cfbeee from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,325 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {cc52e0ba23708df10d696d0e33d9d137 state=OPEN, ts=1447216276325, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,325 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Transition >>>> {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>>> {21b24baeaf26e12d02f79f243076850a state=OPEN, ts=1447216276325, server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>>> 2015-11-10 22:31:16,326 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> cc52e0ba23708df10d696d0e33d9d137 from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:31:16,326 INFO >>>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>>> 21b24baeaf26e12d02f79f243076850a from >>>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>>> 2015-11-10 22:36:01,090 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 22:41:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 22:46:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 22:47:34,680 INFO >>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>>> 10.125.122.237 set balanceSwitch=false >>>> 2015-11-10 22:47:41,845 INFO >>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>>> 10.125.122.237 set balanceSwitch=true >>>> 2015-11-10 22:51:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 22:56:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:01:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:06:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:11:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:16:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:17:34,662 INFO >>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>>> 10.125.122.237 set balanceSwitch=false >>>> 2015-11-10 23:17:38,707 INFO >>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>>> 10.125.122.237 set balanceSwitch=true >>>> 2015-11-10 23:21:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:26:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:31:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:31:01,096 INFO org.apache.hadoop.hbase.mob.MobFileCache: >>>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted >>>> files: 0 >>>> 2015-11-10 23:36:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:41:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:46:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:47:34,668 INFO >>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>>> 10.125.122.237 set balanceSwitch=false >>>> 2015-11-10 23:47:44,843 INFO >>>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>>> 10.125.122.237 set balanceSwitch=true >>>> 2015-11-10 23:51:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0 >>>> 2015-11-10 23:56:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0 >>>> 2015-11-11 00:01:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0 >>>> 2015-11-11 00:06:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0 >>>> 2015-11-11 00:11:01,089 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>>> cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0 >>>> >>>> Region server logs >>>> tail -100 >>>> hbase-cmf-hbase-REGIONSERVER-hadoop4-private.wdc01.infra.ripple.com.log.out >>>> 2015-11-10 22:31:16,255 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,255 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>>> Updated row prod_control,,1434077692105.a4f70c6c01f31f6d3d65e0cf55200c80. >>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>>> Updated row >>>> prod_agg_account_payments,20140325000000|rH48e8bf5kNWuFRBwgm5sCH133cd9zgV29,1447197624013.c2a9f77791512281e4491e0797e824e8. >>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>>> 2015-11-10 22:31:16,259 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,259 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,259 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,259 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,262 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,262 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,265 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,265 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,265 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,265 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,267 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,268 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,272 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,273 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,274 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>>> 1fbbd37bdff125be32d352c98dffd2b1; next sequenceid=180083 >>>> 2015-11-10 22:31:16,274 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>>> 7ce823a4e0f56bf8c854175409bb17b6; next sequenceid=21535 >>>> 2015-11-10 22:31:16,275 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>>> for >>>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1. >>>> 2015-11-10 22:31:16,276 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>>> for >>>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6. >>>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>>> Updated row >>>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1. >>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>>> Updated row >>>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6. >>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>>> 2015-11-10 22:31:16,282 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,282 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,282 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>>> 15701b8131ca505f6cce148d49cfbeee; next sequenceid=1029 >>>> 2015-11-10 22:31:16,283 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,283 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,284 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>>> for >>>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee. >>>> 2015-11-10 22:31:16,304 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>>> Updated row >>>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee. >>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>>> 2015-11-10 22:31:16,308 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,308 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,308 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:16,309 INFO >>>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>>> major jitter 0.500000, min locality to compact 0.000000 >>>> 2015-11-10 22:31:16,318 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>>> 21b24baeaf26e12d02f79f243076850a; next sequenceid=114628 >>>> 2015-11-10 22:31:16,318 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>>> cc52e0ba23708df10d696d0e33d9d137; next sequenceid=161394 >>>> 2015-11-10 22:31:16,320 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>>> for >>>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a. >>>> 2015-11-10 22:31:16,320 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>>> for prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|2014111212 >>>> ,1447198690089.cc52e0ba23708df10d696d0e33d9d137. >>>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>>> Updated row prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD| >>>> 2014111212,1447198690089.cc52e0ba23708df10d696d0e33d9d137. with server= >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>>> Updated row >>>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a. >>>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>>> 2015-11-10 22:31:21,184 WARN org.apache.hadoop.metrics2.lib.Interns: >>>> Metrics intern cache overflow at 2011 for >>>> RegionServer={RegionServer=MetricsInfoImpl{name=RegionServer, >>>> description=RegionServer}, RegionServer >>>> record=MetricsInfoImpl{name=RegionServer, description=RegionServer record}} >>>> 2015-11-10 22:31:22,377 WARN >>>> org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: >>>> hbase.regionserver.global.memstore.upperLimit is deprecated by >>>> hbase.regionserver.global.memstore.size >>>> 2015-11-10 22:31:22,403 INFO >>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 >>>> (all) file(s) in f of >>>> prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041. >>>> into d98861e6b5364d819d11c3f3e1c8fb56(size=180.4 M), total size for store >>>> is 180.4 M. This selection was in queue for 0sec, and took 7sec to execute. >>>> 2015-11-10 22:31:22,405 INFO >>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed >>>> compaction: Request = >>>> regionName=prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041., >>>> storeName=f, fileCount=3, fileSize=180.4 M, priority=27, >>>> time=741524299592168; duration=7sec >>>> 2015-11-10 22:31:22,413 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in >>>> region >>>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d. >>>> 2015-11-10 22:31:22,413 INFO >>>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3 >>>> file(s) in f of >>>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d. >>>> into tmpdir=hdfs:// >>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_transactions/72f5a6c7dba1e486eae6b75f4e9f0d5d/.tmp, >>>> totalSize=82.2 M >>>> 2015-11-10 22:31:22,416 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:25,301 INFO >>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 >>>> (all) file(s) in f of >>>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d. >>>> into f7678de8dd8149428938a0aedfcbcaf9(size=82.2 M), total size for store is >>>> 82.2 M. This selection was in queue for 0sec, and took 2sec to execute. >>>> 2015-11-10 22:31:25,302 INFO >>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed >>>> compaction: Request = >>>> regionName=prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d., >>>> storeName=f, fileCount=3, fileSize=82.2 M, priority=27, >>>> time=741532147583369; duration=2sec >>>> 2015-11-10 22:31:25,305 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in >>>> region >>>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927. >>>> 2015-11-10 22:31:25,305 INFO >>>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3 >>>> file(s) in f of >>>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927. >>>> into tmpdir=hdfs:// >>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_offers/05284b692219df79852f3deaa2ce1927/.tmp, >>>> totalSize=218.4 M >>>> 2015-11-10 22:31:25,310 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:39,481 INFO >>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 >>>> (all) file(s) in f of >>>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927. >>>> into 4aea6e4301f848ff9bf727f4e511978a(size=214.6 M), total size for store >>>> is 214.6 M. This selection was in queue for 0sec, and took 14sec to execute. >>>> 2015-11-10 22:31:39,481 INFO >>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed >>>> compaction: Request = >>>> regionName=prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927., >>>> storeName=f, fileCount=3, fileSize=218.4 M, priority=27, >>>> time=741535039619615; duration=14sec >>>> 2015-11-10 22:31:39,484 INFO >>>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on d in >>>> region >>>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b. >>>> 2015-11-10 22:31:39,484 INFO >>>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3 >>>> file(s) in d of >>>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b. >>>> into tmpdir=hdfs:// >>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_balance_changes/40d39088cc3efe2d230a7d6f8d7b205b/.tmp, >>>> totalSize=197.7 M >>>> 2015-11-10 22:31:39,490 INFO >>>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>>> cacheDataCompressed=false, prefetchOnOpen=false >>>> 2015-11-10 22:31:52,393 INFO >>>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 >>>> (all) file(s) in d of >>>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b. >>>> into 1e48a57cdb544e898e78c9e8276ec05b(size=197.7 M), total size for store >>>> is 197.7 M. This selection was in queue for 0sec, and took 12sec to execute. >>>> 2015-11-10 22:31:52,393 INFO >>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed >>>> compaction: Request = >>>> regionName=prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b., >>>> storeName=d, fileCount=3, fileSize=197.7 M, priority=27, >>>> time=741549219351203; duration=12sec >>>> 2015-11-10 22:32:01,402 INFO >>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Atomically >>>> moving hadoop3-private.wdc01.infra.ripple.com,60020,1447102836904's >>>> wals to my queue >>>> 2015-11-10 22:32:01,405 WARN >>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer 1 didn't >>>> exist, skipping the replay >>>> 2015-11-10 22:32:01,406 WARN >>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer >>>> 1-hadoop4-private.wdc01.infra.ripple.com,60020,1447102248964 didn't >>>> exist, skipping the replay >>>> 2015-11-10 22:32:01,412 WARN >>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Got exception >>>> in copyQueuesFromRSUsingMulti: >>>> org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode >>>> = Directory not empty >>>> at >>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:125) >>>> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1152) >>>> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1118) >>>> at >>>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:685) >>>> at >>>> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1550) >>>> at >>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.copyQueuesFromRSUsingMulti(ReplicationQueuesZKImpl.java:340) >>>> at >>>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.claimQueues(ReplicationQueuesZKImpl.java:172) >>>> at >>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:570) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> 2015-11-10 22:36:01,223 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=3.78 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=4, accesses=63330, hits=6, >>>> hitRatio=0.01%, , cachingAccesses=21, cachingHits=4, >>>> cachingHitsRatio=19.05%, evictions=29, evicted=3, >>>> evictedPerRun=0.1034482792019844 >>>> 2015-11-10 22:41:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.15 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=7, accesses=63856, hits=8, >>>> hitRatio=0.01%, , cachingAccesses=32, cachingHits=6, >>>> cachingHitsRatio=18.75%, evictions=59, evicted=3, >>>> evictedPerRun=0.050847455859184265 >>>> 2015-11-10 22:46:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11, >>>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9, >>>> cachingHitsRatio=21.43%, evictions=89, evicted=3, >>>> evictedPerRun=0.033707864582538605 >>>> 2015-11-10 22:51:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11, >>>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9, >>>> cachingHitsRatio=21.43%, evictions=119, evicted=3, >>>> evictedPerRun=0.02521008439362049 >>>> 2015-11-10 22:56:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11, >>>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9, >>>> cachingHitsRatio=21.43%, evictions=149, evicted=3, >>>> evictedPerRun=0.020134227350354195 >>>> 2015-11-10 23:01:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63872, hits=13, >>>> hitRatio=0.02%, , cachingAccesses=48, cachingHits=11, >>>> cachingHitsRatio=22.92%, evictions=179, evicted=3, >>>> evictedPerRun=0.016759777441620827 >>>> 2015-11-10 23:06:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63890, hits=21, >>>> hitRatio=0.03%, , cachingAccesses=66, cachingHits=19, >>>> cachingHitsRatio=28.79%, evictions=209, evicted=3, >>>> evictedPerRun=0.014354066923260689 >>>> 2015-11-10 23:11:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64416, hits=26, >>>> hitRatio=0.04%, , cachingAccesses=77, cachingHits=24, >>>> cachingHitsRatio=31.17%, evictions=239, evicted=3, >>>> evictedPerRun=0.012552301399409771 >>>> 2015-11-10 23:16:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31, >>>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29, >>>> cachingHitsRatio=33.33%, evictions=269, evicted=3, >>>> evictedPerRun=0.011152416467666626 >>>> 2015-11-10 23:21:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31, >>>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29, >>>> cachingHitsRatio=33.33%, evictions=299, evicted=3, >>>> evictedPerRun=0.010033444501459599 >>>> 2015-11-10 23:26:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31, >>>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29, >>>> cachingHitsRatio=33.33%, evictions=329, evicted=3, >>>> evictedPerRun=0.009118541143834591 >>>> 2015-11-10 23:31:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64432, hits=34, >>>> hitRatio=0.05%, , cachingAccesses=93, cachingHits=32, >>>> cachingHitsRatio=34.41%, evictions=359, evicted=3, >>>> evictedPerRun=0.008356546051800251 >>>> 2015-11-10 23:31:01,228 INFO org.apache.hadoop.hbase.mob.MobFileCache: >>>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted >>>> files: 0 >>>> 2015-11-10 23:31:06,790 INFO >>>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL /hbase/WALs/ >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/ >>>> hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503 >>>> with entries=247, filesize=125.29 KB; new WAL /hbase/WALs/ >>>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/ >>>> hadoop4-private.wdc01.infra.ripple.com >>>> %2C60020%2C1447216261115.null0.1447219866745 >>>> 2015-11-10 23:31:06,792 INFO >>>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving hdfs:// >>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/WALs/hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503 >>>> to hdfs:// >>>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/oldWALs/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503 >>>> 2015-11-10 23:36:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64450, hits=42, >>>> hitRatio=0.07%, , cachingAccesses=111, cachingHits=40, >>>> cachingHitsRatio=36.04%, evictions=389, evicted=3, >>>> evictedPerRun=0.007712082471698523 >>>> 2015-11-10 23:41:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64976, hits=47, >>>> hitRatio=0.07%, , cachingAccesses=122, cachingHits=45, >>>> cachingHitsRatio=36.89%, evictions=419, evicted=3, >>>> evictedPerRun=0.0071599045768380165 >>>> 2015-11-10 23:46:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52, >>>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50, >>>> cachingHitsRatio=37.88%, evictions=449, evicted=3, >>>> evictedPerRun=0.00668151443824172 >>>> 2015-11-10 23:51:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52, >>>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50, >>>> cachingHitsRatio=37.88%, evictions=479, evicted=3, >>>> evictedPerRun=0.00626304792240262 >>>> 2015-11-10 23:56:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52, >>>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50, >>>> cachingHitsRatio=37.88%, evictions=509, evicted=3, >>>> evictedPerRun=0.005893909838050604 >>>> 2015-11-11 00:01:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64992, hits=55, >>>> hitRatio=0.08%, , cachingAccesses=138, cachingHits=53, >>>> cachingHitsRatio=38.41%, evictions=539, evicted=3, >>>> evictedPerRun=0.005565862637013197 >>>> 2015-11-11 00:06:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65010, hits=63, >>>> hitRatio=0.10%, , cachingAccesses=156, cachingHits=61, >>>> cachingHitsRatio=39.10%, evictions=569, evicted=3, >>>> evictedPerRun=0.005272407550364733 >>>> 2015-11-11 00:11:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65536, hits=68, >>>> hitRatio=0.10%, , cachingAccesses=167, cachingHits=66, >>>> cachingHitsRatio=39.52%, evictions=599, evicted=3, >>>> evictedPerRun=0.00500834733247757 >>>> 2015-11-11 00:16:01,222 INFO >>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65546, hits=73, >>>> hitRatio=0.11%, , cachingAccesses=177, cachingHits=71, >>>> cachingHitsRatio=40.11%, evictions=629, evicted=3, >>>> evictedPerRun=0.004769475199282169 >>>> >>>> >>>> >>>> >>>> On Tue, Nov 10, 2015 at 8:15 PM, Ted Yu wrote: >>>> >>>>> bq. The region exits are forcing the data to start transferring to >>>>> other regions, >>>>> >>>>> Can you elaborate a bit more on the above ? Did you mean some regions >>>>> were transferred to other region server(s) ? >>>>> >>>>> Can you determined the effective RegionSplitPolicy ? >>>>> >>>>> Snippets of server logs would help illustrate the situation. >>>>> >>>>> Thanks >>>>> >>>>> On Tue, Nov 10, 2015 at 8:09 PM, Abraham Tom >>>>> wrote: >>>>> >>>>>> I am on CDH 5.4.8 hbase 1.0 >>>>>> >>>>>> I am migrating to a new cluster and the data is sourced from CDH 5.3 >>>>>> hbase >>>>>> 0.98.6 >>>>>> >>>>>> I took snapshots from the old cluster exported them to the new >>>>>> cluster - no >>>>>> errors >>>>>> >>>>>> cloned the table, flush, major_compact and balanced the regions >>>>>> >>>>>> the configs are practically identical to the old cluster >>>>>> >>>>>> >>>>>> >>>>>> hbck and fsck all check out with no inconsistencies found >>>>>> >>>>>> >>>>>> >>>>>> yet when we try to load data via thrift the regions start splitting >>>>>> at an >>>>>> outrageous pace. >>>>>> >>>>>> Eventually the regions become offline starting a snowball affect. The >>>>>> region exits are forcing the data to start transferring to other >>>>>> regions, >>>>>> with that creates large GC activity and multiple regions in transition >>>>>> before thrift can no longer execute any writes. This usually happens >>>>>> after >>>>>> a few records. How we put is that we perform rowkey scans before >>>>>> puts to >>>>>> ensure where we last left off. >>>>>> >>>>>> >>>>>> >>>>>> Thrift eventually stops writing due to regions being offline and no >>>>>> longer >>>>>> serving. The regions eventually stop splitting and all the GC >>>>>> eventually >>>>>> complete. Once things settle back down, I am able to perform hbck >>>>>> and fsck >>>>>> and it again comes back with no inconsistencies >>>>>> >>>>>> >>>>>> >>>>>> I need ideas as to what is causing this >>>>>> >>>>>> >>>>>> -- >>>>>> Abraham Tom >>>>>> Email: work2much@gmail.com >>>>>> Phone: 415-515-3621 >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Abraham Tom >>>> Email: work2much@gmail.com >>>> Phone: 415-515-3621 >>>> >>> >>> >> >> >> -- >> Abraham Tom >> Email: work2much@gmail.com >> Phone: 415-515-3621 >> > > -- Abraham Tom Email: work2much@gmail.com Phone: 415-515-3621 --001a11473af62b72c505245ba83f--