Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 744C918E31 for ; Thu, 12 Nov 2015 14:54:23 +0000 (UTC) Received: (qmail 72408 invoked by uid 500); 12 Nov 2015 14:54:21 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 72334 invoked by uid 500); 12 Nov 2015 14:54:21 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 72322 invoked by uid 99); 12 Nov 2015 14:54:21 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Nov 2015 14:54:21 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 871DD180A60 for ; Thu, 12 Nov 2015 14:54:20 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.151 X-Spam-Level: **** X-Spam-Status: No, score=4.151 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, HTML_MESSAGE=3, KAM_LOTSOFHASH=0.25, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id q-wYEVbcj3Wk for ; Thu, 12 Nov 2015 14:53:59 +0000 (UTC) Received: from mail-yk0-f172.google.com (mail-yk0-f172.google.com [209.85.160.172]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 8E540207E3 for ; Thu, 12 Nov 2015 14:53:58 +0000 (UTC) Received: by ykfs79 with SMTP id s79so97301503ykf.1 for ; Thu, 12 Nov 2015 06:53:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=txgbfUORaG+8wm/8p+UFad3/8mAziC6fBsUqImk97GE=; b=Mmmr845UFtcHE+kVdoXl6P/ptYZPfg+WuhsGqCJR/gNECGLN8oPBtHZF6fb0amyfyq Ks5NcrWSDCo4s/oE9LzoMsLqqCw0euVUd8RgqdBr5z6ycDmnLHsdwUDMoUZXCHFHRgdP 8ZakYljxdBr+hmYF/7wQXpxwFzbu2cS+atkBsHa2xkoGx7nCJH03Idviq4GQKNKxp2tB FtBZlejPfTYqyi3fEjy2VbZtk74htFE24q+QtRiuD7KsxtfEWOWMyvNP740lSEQU3kPR tOhvafLoAD3bTj7iPdiR6J/t608M/FbE7A/1QyfUaL0VE8MURu9ps1h4wb1risEqSw6M c4Gw== MIME-Version: 1.0 X-Received: by 10.13.214.17 with SMTP id y17mr16662983ywd.218.1447340030520; Thu, 12 Nov 2015 06:53:50 -0800 (PST) Received: by 10.37.216.145 with HTTP; Thu, 12 Nov 2015 06:53:50 -0800 (PST) In-Reply-To: References: Date: Thu, 12 Nov 2015 06:53:50 -0800 Message-ID: Subject: Re: hbase regions behaving badly From: Ted Yu To: Abraham Tom Cc: "user@hbase.apache.org" Content-Type: multipart/alternative; boundary=94eb2c073c026376660524591c36 --94eb2c073c026376660524591c36 Content-Type: text/plain; charset=UTF-8 bq. 2015-11-11 12:05:35,179 DEBUG org.apache.hadoop.hdfs.BlockReaderFactory: Closed potentially stale remote peer Have you checked hdfs health ? Cheers On Wed, Nov 11, 2015 at 10:10 AM, Abraham Tom wrote: > Thanks > > I also restarted in debug mode > and found that hbase is renewing my lease to other nodes rather quickly > logs here > http://pastebin.com/4EgsXCDd > > > On Wed, Nov 11, 2015 at 6:20 AM, Ted Yu wrote: > >> Please note that IncreasingToUpperBoundRegionSplitPolicy was in effect. >> >> See http://hbase.apache.org/book.html#_custom_split_policies for related >> information (there is link to >> http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.html >> ). >> >> In the future, please consider using pastebin etc. for server log so that >> the message is concise. >> >> Cheers >> >> On Tue, Nov 10, 2015 at 10:17 PM, Abraham Tom >> wrote: >> >>> my hbase-site.xml snippet >>> >>> >>> hbase.mob.sweep.tool.compaction.memstore.flush.size134217728hbase-default.xml >>> >>> hbase.hregion.memstore.flush.size134217728hbase-default.xml >>> >>> hbase.hregion.memstore.block.multiplier4hbase-default.xml >>> >>> hbase.mob.sweep.tool.compaction.memstore.flush.size134217728hbase-default.xml >>> >>> dfs.blockreport.split.threshold1000000hdfs-default.xml >>> >>> hbase.regionserver.regionSplitLimit1000hbase-default.xml >>> >>> hbase.splitlog.manager.timeout120000hbase-site.xml >>> >>> hbase.regionserver.region.split.policyorg.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicyhbase-default.xml >>> >>> mapreduce.job.split.metainfo.maxsize10000000mapred-default.xml >>> >>> >>> >>> >>> >>> and some master logs >>> tail -100 >>> hbase-cmf-hbase-MASTER-hadoop4-private.wdc01.infra.ripple.com.log.out >>> 2015-11-10 22:31:16,205 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {5097e5a15891c375eca53b992584c52e state=PENDING_OPEN, ts=1447216273284, >>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>> {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>> 2015-11-10 22:31:16,206 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> d3b39413ee1bc88a100cd96564c4c420 from >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>> 2015-11-10 22:31:16,206 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {cff8a38c23c3987fbbebec615f45f4ac state=OPENING, ts=1447216276185, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>> {cff8a38c23c3987fbbebec615f45f4ac state=OPEN, ts=1447216276206, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>> 2015-11-10 22:31:16,206 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> bc842e97c8ff7b0f3498b38510a9c1c1 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,206 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {c73147aef08804451c26c1f8099008f2 state=PENDING_OPEN, ts=1447216273284, >>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>> {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>> 2015-11-10 22:31:16,207 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> cff8a38c23c3987fbbebec615f45f4ac from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,208 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {8a671fb436fb0d186b37211d847aa897 state=OPENING, ts=1447216276188, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {8a671fb436fb0d186b37211d847aa897 state=OPEN, ts=1447216276208, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,209 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 8a671fb436fb0d186b37211d847aa897 from >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>> 2015-11-10 22:31:16,209 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {e920fab780cdd6ba183bc50a8b5d8c72 state=PENDING_OPEN, ts=1447216273276, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,210 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {1258fdeaf4f00989a8282e5f9c215d7e state=OPENING, ts=1447216276193, server= >>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} to >>> {1258fdeaf4f00989a8282e5f9c215d7e state=OPEN, ts=1447216276210, server= >>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} >>> 2015-11-10 22:31:16,211 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 1258fdeaf4f00989a8282e5f9c215d7e from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,213 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {02666182c37df5043acdcaca31ceb095 state=OPENING, ts=1447216276184, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>> {02666182c37df5043acdcaca31ceb095 state=OPEN, ts=1447216276213, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>> 2015-11-10 22:31:16,214 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=PENDING_OPEN, ts=1447216273284, >>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>> 2015-11-10 22:31:16,215 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 02666182c37df5043acdcaca31ceb095 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,216 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {4c694bcbe9679747e738a630aec33c3c state=OPENING, ts=1447216276200, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {4c694bcbe9679747e738a630aec33c3c state=OPEN, ts=1447216276216, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,217 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 4c694bcbe9679747e738a630aec33c3c from >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803 >>> 2015-11-10 22:31:16,217 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {dc3169f842a4d19014f040c887b9acff state=PENDING_OPEN, ts=1447216273280, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,219 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {8320a058823353a19f69a10da6827953 state=OPENING, ts=1447216276198, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {8320a058823353a19f69a10da6827953 state=OPEN, ts=1447216276219, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,219 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 8320a058823353a19f69a10da6827953 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,220 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {edfd9220fe6c2a37fe7521101d490941 state=PENDING_OPEN, ts=1447216273142, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,220 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {1937fd49e94d42fe072a29ff7af885bb state=OPENING, ts=1447216276205, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>> {1937fd49e94d42fe072a29ff7af885bb state=OPEN, ts=1447216276220, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>> 2015-11-10 22:31:16,220 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {e4b861ca490c05fa4276e73e629b471a state=OPENING, ts=1447216276204, server= >>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} to >>> {e4b861ca490c05fa4276e73e629b471a state=OPEN, ts=1447216276220, server= >>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189} >>> 2015-11-10 22:31:16,221 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 1937fd49e94d42fe072a29ff7af885bb from >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803 >>> 2015-11-10 22:31:16,221 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {9ecf375c160ffe4616b8c4fb45ea8fba state=PENDING_OPEN, ts=1447216273274, >>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>> 2015-11-10 22:31:16,221 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> e4b861ca490c05fa4276e73e629b471a from >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>> 2015-11-10 22:31:16,222 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {2c7572eeff0a72eceec10e0301d344a7 state=OPENING, ts=1447216276205, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>> {2c7572eeff0a72eceec10e0301d344a7 state=OPEN, ts=1447216276222, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>> 2015-11-10 22:31:16,222 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 2c7572eeff0a72eceec10e0301d344a7 from >>> hadoop3-private.wdc01.infra.ripple.com,60020,1447208454196 >>> 2015-11-10 22:31:16,223 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {cb7e639dbb58b1e47c1da538165c503c state=PENDING_OPEN, ts=1447216273288, >>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>> {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>> 2015-11-10 22:31:16,225 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>> {5097e5a15891c375eca53b992584c52e state=OPEN, ts=1447216276225, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>> 2015-11-10 22:31:16,225 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>> {c73147aef08804451c26c1f8099008f2 state=OPEN, ts=1447216276225, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>> 2015-11-10 22:31:16,226 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {abddf225d895bd30e11205390fedc03e state=OPENING, ts=1447216276205, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>> {abddf225d895bd30e11205390fedc03e state=OPEN, ts=1447216276226, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>> 2015-11-10 22:31:16,226 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 5097e5a15891c375eca53b992584c52e from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,226 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> c73147aef08804451c26c1f8099008f2 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,227 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> abddf225d895bd30e11205390fedc03e from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,227 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {a5e6796ae343481b92b0180d817c1e17 state=PENDING_OPEN, ts=1447216273288, >>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>> {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>> 2015-11-10 22:31:16,229 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} to >>> {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPEN, ts=1447216276229, server= >>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212} >>> 2015-11-10 22:31:16,230 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {e920fab780cdd6ba183bc50a8b5d8c72 state=OPEN, ts=1447216276230, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,230 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> dae9fb9b19a8d2076fd4be2b95aefe45 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,230 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> e920fab780cdd6ba183bc50a8b5d8c72 from >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>> 2015-11-10 22:31:16,230 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {2c04351380cc266df609ab53ef0e56ad state=PENDING_OPEN, ts=1447216273276, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,236 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {edfd9220fe6c2a37fe7521101d490941 state=OPEN, ts=1447216276236, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,236 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>> {9ecf375c160ffe4616b8c4fb45ea8fba state=OPEN, ts=1447216276236, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>> 2015-11-10 22:31:16,237 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {dc3169f842a4d19014f040c887b9acff state=OPEN, ts=1447216276237, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,240 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> edfd9220fe6c2a37fe7521101d490941 from >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803 >>> 2015-11-10 22:31:16,240 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=PENDING_OPEN, ts=1447216273280, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,241 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {c2a9f77791512281e4491e0797e824e8 state=PENDING_OPEN, ts=1447216273277, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,241 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> dc3169f842a4d19014f040c887b9acff from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,241 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 9ecf375c160ffe4616b8c4fb45ea8fba from >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>> 2015-11-10 22:31:16,244 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>> {cb7e639dbb58b1e47c1da538165c503c state=OPEN, ts=1447216276244, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>> 2015-11-10 22:31:16,245 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> cb7e639dbb58b1e47c1da538165c503c from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,246 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} to >>> {a5e6796ae343481b92b0180d817c1e17 state=OPEN, ts=1447216276246, server= >>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019} >>> 2015-11-10 22:31:16,246 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> a5e6796ae343481b92b0180d817c1e17 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,252 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {2c04351380cc266df609ab53ef0e56ad state=OPEN, ts=1447216276252, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,253 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 2c04351380cc266df609ab53ef0e56ad from >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>> 2015-11-10 22:31:16,253 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {15701b8131ca505f6cce148d49cfbeee state=PENDING_OPEN, ts=1447216273280, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,257 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPEN, ts=1447216276257, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,257 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {c2a9f77791512281e4491e0797e824e8 state=OPEN, ts=1447216276257, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,258 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> a4f70c6c01f31f6d3d65e0cf55200c80 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,258 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> c2a9f77791512281e4491e0797e824e8 from >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227 >>> 2015-11-10 22:31:16,258 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {7ce823a4e0f56bf8c854175409bb17b6 state=PENDING_OPEN, ts=1447216273280, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,258 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {1fbbd37bdff125be32d352c98dffd2b1 state=PENDING_OPEN, ts=1447216273280, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,280 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {1fbbd37bdff125be32d352c98dffd2b1 state=OPEN, ts=1447216276280, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,280 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {7ce823a4e0f56bf8c854175409bb17b6 state=OPEN, ts=1447216276280, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,281 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 1fbbd37bdff125be32d352c98dffd2b1 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,281 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 7ce823a4e0f56bf8c854175409bb17b6 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,281 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {cc52e0ba23708df10d696d0e33d9d137 state=PENDING_OPEN, ts=1447216273280, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,281 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {21b24baeaf26e12d02f79f243076850a state=PENDING_OPEN, ts=1447216273280, >>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,306 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {15701b8131ca505f6cce148d49cfbeee state=OPEN, ts=1447216276306, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,307 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 15701b8131ca505f6cce148d49cfbeee from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,325 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {cc52e0ba23708df10d696d0e33d9d137 state=OPEN, ts=1447216276325, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,325 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Transition >>> {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} to >>> {21b24baeaf26e12d02f79f243076850a state=OPEN, ts=1447216276325, server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115} >>> 2015-11-10 22:31:16,326 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> cc52e0ba23708df10d696d0e33d9d137 from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:31:16,326 INFO >>> org.apache.hadoop.hbase.master.RegionStates: Offlined >>> 21b24baeaf26e12d02f79f243076850a from >>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361 >>> 2015-11-10 22:36:01,090 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 22:41:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 22:46:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 22:47:34,680 INFO >>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>> 10.125.122.237 set balanceSwitch=false >>> 2015-11-10 22:47:41,845 INFO >>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>> 10.125.122.237 set balanceSwitch=true >>> 2015-11-10 22:51:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 22:56:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:01:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:06:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:11:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:16:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:17:34,662 INFO >>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>> 10.125.122.237 set balanceSwitch=false >>> 2015-11-10 23:17:38,707 INFO >>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>> 10.125.122.237 set balanceSwitch=true >>> 2015-11-10 23:21:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:26:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:31:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:31:01,096 INFO org.apache.hadoop.hbase.mob.MobFileCache: >>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted >>> files: 0 >>> 2015-11-10 23:36:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:41:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:46:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:47:34,668 INFO >>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>> 10.125.122.237 set balanceSwitch=false >>> 2015-11-10 23:47:44,843 INFO >>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase// >>> 10.125.122.237 set balanceSwitch=true >>> 2015-11-10 23:51:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0 >>> 2015-11-10 23:56:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0 >>> 2015-11-11 00:01:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0 >>> 2015-11-11 00:06:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0 >>> 2015-11-11 00:11:01,089 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB, >>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0, >>> hitRatio=0, cachingAccesses=0, cachingHits=0, >>> cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0 >>> >>> Region server logs >>> tail -100 >>> hbase-cmf-hbase-REGIONSERVER-hadoop4-private.wdc01.infra.ripple.com.log.out >>> 2015-11-10 22:31:16,255 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,255 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>> Updated row prod_control,,1434077692105.a4f70c6c01f31f6d3d65e0cf55200c80. >>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>> Updated row >>> prod_agg_account_payments,20140325000000|rH48e8bf5kNWuFRBwgm5sCH133cd9zgV29,1447197624013.c2a9f77791512281e4491e0797e824e8. >>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>> 2015-11-10 22:31:16,259 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,259 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,259 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,259 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,262 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,262 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,265 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,265 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,265 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,265 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,267 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,268 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,272 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,273 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,274 INFO >>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>> 1fbbd37bdff125be32d352c98dffd2b1; next sequenceid=180083 >>> 2015-11-10 22:31:16,274 INFO >>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>> 7ce823a4e0f56bf8c854175409bb17b6; next sequenceid=21535 >>> 2015-11-10 22:31:16,275 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>> for >>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1. >>> 2015-11-10 22:31:16,276 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>> for >>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6. >>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>> Updated row >>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1. >>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>> Updated row >>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6. >>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>> 2015-11-10 22:31:16,282 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,282 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,282 INFO >>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>> 15701b8131ca505f6cce148d49cfbeee; next sequenceid=1029 >>> 2015-11-10 22:31:16,283 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,283 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,284 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>> for >>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee. >>> 2015-11-10 22:31:16,304 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>> Updated row >>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee. >>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>> 2015-11-10 22:31:16,308 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,308 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,308 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:16,309 INFO >>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration: >>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000; >>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000, >>> major jitter 0.500000, min locality to compact 0.000000 >>> 2015-11-10 22:31:16,318 INFO >>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>> 21b24baeaf26e12d02f79f243076850a; next sequenceid=114628 >>> 2015-11-10 22:31:16,318 INFO >>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined >>> cc52e0ba23708df10d696d0e33d9d137; next sequenceid=161394 >>> 2015-11-10 22:31:16,320 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>> for >>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a. >>> 2015-11-10 22:31:16,320 INFO >>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks >>> for prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|2014111212 >>> ,1447198690089.cc52e0ba23708df10d696d0e33d9d137. >>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>> Updated row prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD| >>> 2014111212,1447198690089.cc52e0ba23708df10d696d0e33d9d137. with server= >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor: >>> Updated row >>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a. >>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115 >>> 2015-11-10 22:31:21,184 WARN org.apache.hadoop.metrics2.lib.Interns: >>> Metrics intern cache overflow at 2011 for >>> RegionServer={RegionServer=MetricsInfoImpl{name=RegionServer, >>> description=RegionServer}, RegionServer >>> record=MetricsInfoImpl{name=RegionServer, description=RegionServer record}} >>> 2015-11-10 22:31:22,377 WARN >>> org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil: >>> hbase.regionserver.global.memstore.upperLimit is deprecated by >>> hbase.regionserver.global.memstore.size >>> 2015-11-10 22:31:22,403 INFO >>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 >>> (all) file(s) in f of >>> prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041. >>> into d98861e6b5364d819d11c3f3e1c8fb56(size=180.4 M), total size for store >>> is 180.4 M. This selection was in queue for 0sec, and took 7sec to execute. >>> 2015-11-10 22:31:22,405 INFO >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed >>> compaction: Request = >>> regionName=prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041., >>> storeName=f, fileCount=3, fileSize=180.4 M, priority=27, >>> time=741524299592168; duration=7sec >>> 2015-11-10 22:31:22,413 INFO >>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in >>> region >>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d. >>> 2015-11-10 22:31:22,413 INFO >>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3 >>> file(s) in f of >>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d. >>> into tmpdir=hdfs:// >>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_transactions/72f5a6c7dba1e486eae6b75f4e9f0d5d/.tmp, >>> totalSize=82.2 M >>> 2015-11-10 22:31:22,416 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:25,301 INFO >>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 >>> (all) file(s) in f of >>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d. >>> into f7678de8dd8149428938a0aedfcbcaf9(size=82.2 M), total size for store is >>> 82.2 M. This selection was in queue for 0sec, and took 2sec to execute. >>> 2015-11-10 22:31:25,302 INFO >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed >>> compaction: Request = >>> regionName=prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d., >>> storeName=f, fileCount=3, fileSize=82.2 M, priority=27, >>> time=741532147583369; duration=2sec >>> 2015-11-10 22:31:25,305 INFO >>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in >>> region >>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927. >>> 2015-11-10 22:31:25,305 INFO >>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3 >>> file(s) in f of >>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927. >>> into tmpdir=hdfs:// >>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_offers/05284b692219df79852f3deaa2ce1927/.tmp, >>> totalSize=218.4 M >>> 2015-11-10 22:31:25,310 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:39,481 INFO >>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 >>> (all) file(s) in f of >>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927. >>> into 4aea6e4301f848ff9bf727f4e511978a(size=214.6 M), total size for store >>> is 214.6 M. This selection was in queue for 0sec, and took 14sec to execute. >>> 2015-11-10 22:31:39,481 INFO >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed >>> compaction: Request = >>> regionName=prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927., >>> storeName=f, fileCount=3, fileSize=218.4 M, priority=27, >>> time=741535039619615; duration=14sec >>> 2015-11-10 22:31:39,484 INFO >>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on d in >>> region >>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b. >>> 2015-11-10 22:31:39,484 INFO >>> org.apache.hadoop.hbase.regionserver.HStore: Starting compaction of 3 >>> file(s) in d of >>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b. >>> into tmpdir=hdfs:// >>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_balance_changes/40d39088cc3efe2d230a7d6f8d7b205b/.tmp, >>> totalSize=197.7 M >>> 2015-11-10 22:31:39,490 INFO >>> org.apache.hadoop.hbase.io.hfile.CacheConfig: >>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize= >>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112, >>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5, >>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false, >>> cacheDataOnWrite=false, cacheIndexesOnWrite=false, >>> cacheBloomsOnWrite=false, cacheEvictOnClose=false, >>> cacheDataCompressed=false, prefetchOnOpen=false >>> 2015-11-10 22:31:52,393 INFO >>> org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 3 >>> (all) file(s) in d of >>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b. >>> into 1e48a57cdb544e898e78c9e8276ec05b(size=197.7 M), total size for store >>> is 197.7 M. This selection was in queue for 0sec, and took 12sec to execute. >>> 2015-11-10 22:31:52,393 INFO >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed >>> compaction: Request = >>> regionName=prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b., >>> storeName=d, fileCount=3, fileSize=197.7 M, priority=27, >>> time=741549219351203; duration=12sec >>> 2015-11-10 22:32:01,402 INFO >>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Atomically >>> moving hadoop3-private.wdc01.infra.ripple.com,60020,1447102836904's >>> wals to my queue >>> 2015-11-10 22:32:01,405 WARN >>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer 1 didn't >>> exist, skipping the replay >>> 2015-11-10 22:32:01,406 WARN >>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer >>> 1-hadoop4-private.wdc01.infra.ripple.com,60020,1447102248964 didn't >>> exist, skipping the replay >>> 2015-11-10 22:32:01,412 WARN >>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Got exception >>> in copyQueuesFromRSUsingMulti: >>> org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode >>> = Directory not empty >>> at >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:125) >>> at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1152) >>> at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1118) >>> at >>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:685) >>> at >>> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1550) >>> at >>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.copyQueuesFromRSUsingMulti(ReplicationQueuesZKImpl.java:340) >>> at >>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.claimQueues(ReplicationQueuesZKImpl.java:172) >>> at >>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:570) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> 2015-11-10 22:36:01,223 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=3.78 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=4, accesses=63330, hits=6, >>> hitRatio=0.01%, , cachingAccesses=21, cachingHits=4, >>> cachingHitsRatio=19.05%, evictions=29, evicted=3, >>> evictedPerRun=0.1034482792019844 >>> 2015-11-10 22:41:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.15 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=7, accesses=63856, hits=8, >>> hitRatio=0.01%, , cachingAccesses=32, cachingHits=6, >>> cachingHitsRatio=18.75%, evictions=59, evicted=3, >>> evictedPerRun=0.050847455859184265 >>> 2015-11-10 22:46:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11, >>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9, >>> cachingHitsRatio=21.43%, evictions=89, evicted=3, >>> evictedPerRun=0.033707864582538605 >>> 2015-11-10 22:51:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11, >>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9, >>> cachingHitsRatio=21.43%, evictions=119, evicted=3, >>> evictedPerRun=0.02521008439362049 >>> 2015-11-10 22:56:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11, >>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9, >>> cachingHitsRatio=21.43%, evictions=149, evicted=3, >>> evictedPerRun=0.020134227350354195 >>> 2015-11-10 23:01:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63872, hits=13, >>> hitRatio=0.02%, , cachingAccesses=48, cachingHits=11, >>> cachingHitsRatio=22.92%, evictions=179, evicted=3, >>> evictedPerRun=0.016759777441620827 >>> 2015-11-10 23:06:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63890, hits=21, >>> hitRatio=0.03%, , cachingAccesses=66, cachingHits=19, >>> cachingHitsRatio=28.79%, evictions=209, evicted=3, >>> evictedPerRun=0.014354066923260689 >>> 2015-11-10 23:11:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64416, hits=26, >>> hitRatio=0.04%, , cachingAccesses=77, cachingHits=24, >>> cachingHitsRatio=31.17%, evictions=239, evicted=3, >>> evictedPerRun=0.012552301399409771 >>> 2015-11-10 23:16:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31, >>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29, >>> cachingHitsRatio=33.33%, evictions=269, evicted=3, >>> evictedPerRun=0.011152416467666626 >>> 2015-11-10 23:21:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31, >>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29, >>> cachingHitsRatio=33.33%, evictions=299, evicted=3, >>> evictedPerRun=0.010033444501459599 >>> 2015-11-10 23:26:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31, >>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29, >>> cachingHitsRatio=33.33%, evictions=329, evicted=3, >>> evictedPerRun=0.009118541143834591 >>> 2015-11-10 23:31:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64432, hits=34, >>> hitRatio=0.05%, , cachingAccesses=93, cachingHits=32, >>> cachingHitsRatio=34.41%, evictions=359, evicted=3, >>> evictedPerRun=0.008356546051800251 >>> 2015-11-10 23:31:01,228 INFO org.apache.hadoop.hbase.mob.MobFileCache: >>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted >>> files: 0 >>> 2015-11-10 23:31:06,790 INFO >>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL /hbase/WALs/ >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/ >>> hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503 >>> with entries=247, filesize=125.29 KB; new WAL /hbase/WALs/ >>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/ >>> hadoop4-private.wdc01.infra.ripple.com >>> %2C60020%2C1447216261115.null0.1447219866745 >>> 2015-11-10 23:31:06,792 INFO >>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving hdfs:// >>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/WALs/hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503 >>> to hdfs:// >>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/oldWALs/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503 >>> 2015-11-10 23:36:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64450, hits=42, >>> hitRatio=0.07%, , cachingAccesses=111, cachingHits=40, >>> cachingHitsRatio=36.04%, evictions=389, evicted=3, >>> evictedPerRun=0.007712082471698523 >>> 2015-11-10 23:41:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64976, hits=47, >>> hitRatio=0.07%, , cachingAccesses=122, cachingHits=45, >>> cachingHitsRatio=36.89%, evictions=419, evicted=3, >>> evictedPerRun=0.0071599045768380165 >>> 2015-11-10 23:46:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52, >>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50, >>> cachingHitsRatio=37.88%, evictions=449, evicted=3, >>> evictedPerRun=0.00668151443824172 >>> 2015-11-10 23:51:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52, >>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50, >>> cachingHitsRatio=37.88%, evictions=479, evicted=3, >>> evictedPerRun=0.00626304792240262 >>> 2015-11-10 23:56:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52, >>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50, >>> cachingHitsRatio=37.88%, evictions=509, evicted=3, >>> evictedPerRun=0.005893909838050604 >>> 2015-11-11 00:01:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64992, hits=55, >>> hitRatio=0.08%, , cachingAccesses=138, cachingHits=53, >>> cachingHitsRatio=38.41%, evictions=539, evicted=3, >>> evictedPerRun=0.005565862637013197 >>> 2015-11-11 00:06:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65010, hits=63, >>> hitRatio=0.10%, , cachingAccesses=156, cachingHits=61, >>> cachingHitsRatio=39.10%, evictions=569, evicted=3, >>> evictedPerRun=0.005272407550364733 >>> 2015-11-11 00:11:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65536, hits=68, >>> hitRatio=0.10%, , cachingAccesses=167, cachingHits=66, >>> cachingHitsRatio=39.52%, evictions=599, evicted=3, >>> evictedPerRun=0.00500834733247757 >>> 2015-11-11 00:16:01,222 INFO >>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB, >>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65546, hits=73, >>> hitRatio=0.11%, , cachingAccesses=177, cachingHits=71, >>> cachingHitsRatio=40.11%, evictions=629, evicted=3, >>> evictedPerRun=0.004769475199282169 >>> >>> >>> >>> >>> On Tue, Nov 10, 2015 at 8:15 PM, Ted Yu wrote: >>> >>>> bq. The region exits are forcing the data to start transferring to >>>> other regions, >>>> >>>> Can you elaborate a bit more on the above ? Did you mean some regions >>>> were transferred to other region server(s) ? >>>> >>>> Can you determined the effective RegionSplitPolicy ? >>>> >>>> Snippets of server logs would help illustrate the situation. >>>> >>>> Thanks >>>> >>>> On Tue, Nov 10, 2015 at 8:09 PM, Abraham Tom >>>> wrote: >>>> >>>>> I am on CDH 5.4.8 hbase 1.0 >>>>> >>>>> I am migrating to a new cluster and the data is sourced from CDH 5.3 >>>>> hbase >>>>> 0.98.6 >>>>> >>>>> I took snapshots from the old cluster exported them to the new cluster >>>>> - no >>>>> errors >>>>> >>>>> cloned the table, flush, major_compact and balanced the regions >>>>> >>>>> the configs are practically identical to the old cluster >>>>> >>>>> >>>>> >>>>> hbck and fsck all check out with no inconsistencies found >>>>> >>>>> >>>>> >>>>> yet when we try to load data via thrift the regions start splitting at >>>>> an >>>>> outrageous pace. >>>>> >>>>> Eventually the regions become offline starting a snowball affect. The >>>>> region exits are forcing the data to start transferring to other >>>>> regions, >>>>> with that creates large GC activity and multiple regions in transition >>>>> before thrift can no longer execute any writes. This usually happens >>>>> after >>>>> a few records. How we put is that we perform rowkey scans before >>>>> puts to >>>>> ensure where we last left off. >>>>> >>>>> >>>>> >>>>> Thrift eventually stops writing due to regions being offline and no >>>>> longer >>>>> serving. The regions eventually stop splitting and all the GC >>>>> eventually >>>>> complete. Once things settle back down, I am able to perform hbck and >>>>> fsck >>>>> and it again comes back with no inconsistencies >>>>> >>>>> >>>>> >>>>> I need ideas as to what is causing this >>>>> >>>>> >>>>> -- >>>>> Abraham Tom >>>>> Email: work2much@gmail.com >>>>> Phone: 415-515-3621 >>>>> >>>> >>>> >>> >>> >>> -- >>> Abraham Tom >>> Email: work2much@gmail.com >>> Phone: 415-515-3621 >>> >> >> > > > -- > Abraham Tom > Email: work2much@gmail.com > Phone: 415-515-3621 > --94eb2c073c026376660524591c36--