hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abraham Tom <work2m...@gmail.com>
Subject Re: hbase regions behaving badly
Date Wed, 11 Nov 2015 18:10:41 GMT
Thanks

I also restarted in debug mode
and found that hbase is renewing my lease to other nodes rather quickly
logs here
http://pastebin.com/4EgsXCDd


On Wed, Nov 11, 2015 at 6:20 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Please note that IncreasingToUpperBoundRegionSplitPolicy was in effect.
>
> See http://hbase.apache.org/book.html#_custom_split_policies for related
> information (there is link to
> http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.html
> ).
>
> In the future, please consider using pastebin etc. for server log so that
> the message is concise.
>
> Cheers
>
> On Tue, Nov 10, 2015 at 10:17 PM, Abraham Tom <work2much@gmail.com> wrote:
>
>> my hbase-site.xml snippet
>>
>>
>> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value><source>hbase-default.xml</source></property>
>>
>> <property><name>hbase.mob.sweep.tool.compaction.memstore.flush.size</name><value>134217728</value><source>hbase-default.xml</source></property>
>>
>> <property><name>dfs.blockreport.split.threshold</name><value>1000000</value><source>hdfs-default.xml</source></property>
>>
>> <property><name>hbase.regionserver.regionSplitLimit</name><value>1000</value><source>hbase-default.xml</source></property>
>>
>> <property><name>hbase.splitlog.manager.timeout</name><value>120000</value><source>hbase-site.xml</source></property>
>>
>> <property><name>hbase.regionserver.region.split.policy</name><value>org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSplitPolicy</value><source>hbase-default.xml</source></property>
>>
>> <property><name>mapreduce.job.split.metainfo.maxsize</name><value>10000000</value><source>mapred-default.xml</source></property>
>>
>>
>>
>>
>>
>> and some master logs
>> tail -100
>> hbase-cmf-hbase-MASTER-hadoop4-private.wdc01.infra.ripple.com.log.out
>> 2015-11-10 22:31:16,205 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {5097e5a15891c375eca53b992584c52e state=PENDING_OPEN,
>> ts=1447216273284, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {5097e5a15891c375eca53b992584c52e state=OPENING, ts=1447216276205,
>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined d3b39413ee1bc88a100cd96564c4c420 from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cff8a38c23c3987fbbebec615f45f4ac state=OPENING,
>> ts=1447216276185, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {cff8a38c23c3987fbbebec615f45f4ac state=OPEN, ts=1447216276206, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined bc842e97c8ff7b0f3498b38510a9c1c1 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,206 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {c73147aef08804451c26c1f8099008f2 state=PENDING_OPEN,
>> ts=1447216273284, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {c73147aef08804451c26c1f8099008f2 state=OPENING, ts=1447216276206,
>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,207 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined cff8a38c23c3987fbbebec615f45f4ac from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,208 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {8a671fb436fb0d186b37211d847aa897 state=OPENING,
>> ts=1447216276188, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {8a671fb436fb0d186b37211d847aa897 state=OPEN, ts=1447216276208, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,209 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 8a671fb436fb0d186b37211d847aa897 from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,209 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {e920fab780cdd6ba183bc50a8b5d8c72 state=PENDING_OPEN,
>> ts=1447216273276, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING, ts=1447216276209,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,210 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {1258fdeaf4f00989a8282e5f9c215d7e state=OPENING,
>> ts=1447216276193, server=hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>> to {1258fdeaf4f00989a8282e5f9c215d7e state=OPEN, ts=1447216276210, server=
>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>> 2015-11-10 22:31:16,211 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 1258fdeaf4f00989a8282e5f9c215d7e from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,213 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {02666182c37df5043acdcaca31ceb095 state=OPENING,
>> ts=1447216276184, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {02666182c37df5043acdcaca31ceb095 state=OPEN, ts=1447216276213, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,214 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {dae9fb9b19a8d2076fd4be2b95aefe45 state=PENDING_OPEN,
>> ts=1447216273284, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING, ts=1447216276214,
>> server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,215 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 02666182c37df5043acdcaca31ceb095 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,216 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {4c694bcbe9679747e738a630aec33c3c state=OPENING,
>> ts=1447216276200, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {4c694bcbe9679747e738a630aec33c3c state=OPEN, ts=1447216276216, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,217 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 4c694bcbe9679747e738a630aec33c3c from
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>> 2015-11-10 22:31:16,217 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {dc3169f842a4d19014f040c887b9acff state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {dc3169f842a4d19014f040c887b9acff state=OPENING, ts=1447216276217,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,219 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {8320a058823353a19f69a10da6827953 state=OPENING,
>> ts=1447216276198, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {8320a058823353a19f69a10da6827953 state=OPEN, ts=1447216276219, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,219 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 8320a058823353a19f69a10da6827953 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {edfd9220fe6c2a37fe7521101d490941 state=PENDING_OPEN,
>> ts=1447216273142, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {edfd9220fe6c2a37fe7521101d490941 state=OPENING, ts=1447216276220,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {1937fd49e94d42fe072a29ff7af885bb state=OPENING,
>> ts=1447216276205, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {1937fd49e94d42fe072a29ff7af885bb state=OPEN, ts=1447216276220, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,220 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {e4b861ca490c05fa4276e73e629b471a state=OPENING,
>> ts=1447216276204, server=hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>> to {e4b861ca490c05fa4276e73e629b471a state=OPEN, ts=1447216276220, server=
>> hadoop3-private.wdc01.infra.ripple.com,60020,1447216261189}
>> 2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 1937fd49e94d42fe072a29ff7af885bb from
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>> 2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {9ecf375c160ffe4616b8c4fb45ea8fba state=PENDING_OPEN,
>> ts=1447216273274, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING, ts=1447216276221,
>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,221 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined e4b861ca490c05fa4276e73e629b471a from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,222 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {2c7572eeff0a72eceec10e0301d344a7 state=OPENING,
>> ts=1447216276205, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {2c7572eeff0a72eceec10e0301d344a7 state=OPEN, ts=1447216276222, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,222 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 2c7572eeff0a72eceec10e0301d344a7 from
>> hadoop3-private.wdc01.infra.ripple.com,60020,1447208454196
>> 2015-11-10 22:31:16,223 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cb7e639dbb58b1e47c1da538165c503c state=PENDING_OPEN,
>> ts=1447216273288, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {cb7e639dbb58b1e47c1da538165c503c state=OPENING, ts=1447216276223,
>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,225 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {5097e5a15891c375eca53b992584c52e state=OPENING,
>> ts=1447216276205, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {5097e5a15891c375eca53b992584c52e state=OPEN, ts=1447216276225, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,225 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {c73147aef08804451c26c1f8099008f2 state=OPENING,
>> ts=1447216276206, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {c73147aef08804451c26c1f8099008f2 state=OPEN, ts=1447216276225, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {abddf225d895bd30e11205390fedc03e state=OPENING,
>> ts=1447216276205, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {abddf225d895bd30e11205390fedc03e state=OPEN, ts=1447216276226, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 5097e5a15891c375eca53b992584c52e from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,226 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined c73147aef08804451c26c1f8099008f2 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,227 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined abddf225d895bd30e11205390fedc03e from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,227 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {a5e6796ae343481b92b0180d817c1e17 state=PENDING_OPEN,
>> ts=1447216273288, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {a5e6796ae343481b92b0180d817c1e17 state=OPENING, ts=1447216276227,
>> server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,229 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPENING,
>> ts=1447216276214, server=hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> to {dae9fb9b19a8d2076fd4be2b95aefe45 state=OPEN, ts=1447216276229, server=
>> hadoop2-private.wdc01.infra.ripple.com,60020,1447216261212}
>> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {e920fab780cdd6ba183bc50a8b5d8c72 state=OPENING,
>> ts=1447216276209, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {e920fab780cdd6ba183bc50a8b5d8c72 state=OPEN, ts=1447216276230, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined dae9fb9b19a8d2076fd4be2b95aefe45 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined e920fab780cdd6ba183bc50a8b5d8c72 from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,230 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {2c04351380cc266df609ab53ef0e56ad state=PENDING_OPEN,
>> ts=1447216273276, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {2c04351380cc266df609ab53ef0e56ad state=OPENING, ts=1447216276230,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,236 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {edfd9220fe6c2a37fe7521101d490941 state=OPENING,
>> ts=1447216276220, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {edfd9220fe6c2a37fe7521101d490941 state=OPEN, ts=1447216276236, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,236 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {9ecf375c160ffe4616b8c4fb45ea8fba state=OPENING,
>> ts=1447216276221, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {9ecf375c160ffe4616b8c4fb45ea8fba state=OPEN, ts=1447216276236, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,237 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {dc3169f842a4d19014f040c887b9acff state=OPENING,
>> ts=1447216276217, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {dc3169f842a4d19014f040c887b9acff state=OPEN, ts=1447216276237, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,240 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined edfd9220fe6c2a37fe7521101d490941 from
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447208455803
>> 2015-11-10 22:31:16,240 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {a4f70c6c01f31f6d3d65e0cf55200c80 state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING, ts=1447216276240,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {c2a9f77791512281e4491e0797e824e8 state=PENDING_OPEN,
>> ts=1447216273277, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {c2a9f77791512281e4491e0797e824e8 state=OPENING, ts=1447216276241,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined dc3169f842a4d19014f040c887b9acff from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,241 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 9ecf375c160ffe4616b8c4fb45ea8fba from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,244 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cb7e639dbb58b1e47c1da538165c503c state=OPENING,
>> ts=1447216276223, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {cb7e639dbb58b1e47c1da538165c503c state=OPEN, ts=1447216276244, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,245 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined cb7e639dbb58b1e47c1da538165c503c from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,246 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {a5e6796ae343481b92b0180d817c1e17 state=OPENING,
>> ts=1447216276227, server=hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> to {a5e6796ae343481b92b0180d817c1e17 state=OPEN, ts=1447216276246, server=
>> hadoop5-private.wdc01.infra.ripple.com,60020,1447216261019}
>> 2015-11-10 22:31:16,246 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined a5e6796ae343481b92b0180d817c1e17 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,252 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {2c04351380cc266df609ab53ef0e56ad state=OPENING,
>> ts=1447216276230, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {2c04351380cc266df609ab53ef0e56ad state=OPEN, ts=1447216276252, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,253 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 2c04351380cc266df609ab53ef0e56ad from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,253 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {15701b8131ca505f6cce148d49cfbeee state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {15701b8131ca505f6cce148d49cfbeee state=OPENING, ts=1447216276253,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,257 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPENING,
>> ts=1447216276240, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {a4f70c6c01f31f6d3d65e0cf55200c80 state=OPEN, ts=1447216276257, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,257 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {c2a9f77791512281e4491e0797e824e8 state=OPENING,
>> ts=1447216276241, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {c2a9f77791512281e4491e0797e824e8 state=OPEN, ts=1447216276257, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined a4f70c6c01f31f6d3d65e0cf55200c80 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined c2a9f77791512281e4491e0797e824e8 from
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447208436227
>> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {7ce823a4e0f56bf8c854175409bb17b6 state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING, ts=1447216276258,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,258 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {1fbbd37bdff125be32d352c98dffd2b1 state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING, ts=1447216276258,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,280 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {1fbbd37bdff125be32d352c98dffd2b1 state=OPENING,
>> ts=1447216276258, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {1fbbd37bdff125be32d352c98dffd2b1 state=OPEN, ts=1447216276280, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,280 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {7ce823a4e0f56bf8c854175409bb17b6 state=OPENING,
>> ts=1447216276258, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {7ce823a4e0f56bf8c854175409bb17b6 state=OPEN, ts=1447216276280, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 1fbbd37bdff125be32d352c98dffd2b1 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 7ce823a4e0f56bf8c854175409bb17b6 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cc52e0ba23708df10d696d0e33d9d137 state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {cc52e0ba23708df10d696d0e33d9d137 state=OPENING, ts=1447216276281,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,281 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {21b24baeaf26e12d02f79f243076850a state=PENDING_OPEN,
>> ts=1447216273280, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {21b24baeaf26e12d02f79f243076850a state=OPENING, ts=1447216276281,
>> server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,306 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {15701b8131ca505f6cce148d49cfbeee state=OPENING,
>> ts=1447216276253, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {15701b8131ca505f6cce148d49cfbeee state=OPEN, ts=1447216276306, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,307 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 15701b8131ca505f6cce148d49cfbeee from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,325 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {cc52e0ba23708df10d696d0e33d9d137 state=OPENING,
>> ts=1447216276281, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {cc52e0ba23708df10d696d0e33d9d137 state=OPEN, ts=1447216276325, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,325 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Transition {21b24baeaf26e12d02f79f243076850a state=OPENING,
>> ts=1447216276281, server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> to {21b24baeaf26e12d02f79f243076850a state=OPEN, ts=1447216276325, server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115}
>> 2015-11-10 22:31:16,326 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined cc52e0ba23708df10d696d0e33d9d137 from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:31:16,326 INFO org.apache.hadoop.hbase.master.RegionStates:
>> Offlined 21b24baeaf26e12d02f79f243076850a from
>> hadoop1-private.wdc01.infra.ripple.com,60020,1447208527361
>> 2015-11-10 22:36:01,090 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0
>> 2015-11-10 22:41:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=59, evicted=0, evictedPerRun=0.0
>> 2015-11-10 22:46:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0
>> 2015-11-10 22:47:34,680 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=false
>> 2015-11-10 22:47:41,845 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=true
>> 2015-11-10 22:51:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
>> 2015-11-10 22:56:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:01:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=179, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:06:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=209, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:11:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=239, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:16:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=269, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:17:34,662 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=false
>> 2015-11-10 23:17:38,707 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=true
>> 2015-11-10 23:21:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=299, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:26:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=329, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:31:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=359, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:31:01,096 INFO org.apache.hadoop.hbase.mob.MobFileCache:
>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
>> files: 0
>> 2015-11-10 23:36:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=389, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:41:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=419, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:46:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=449, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:47:34,668 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=false
>> 2015-11-10 23:47:44,843 INFO
>> org.apache.hadoop.hbase.master.MasterRpcServices: Client=hbase//
>> 10.125.122.237 set balanceSwitch=true
>> 2015-11-10 23:51:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=479, evicted=0, evictedPerRun=0.0
>> 2015-11-10 23:56:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=509, evicted=0, evictedPerRun=0.0
>> 2015-11-11 00:01:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=539, evicted=0, evictedPerRun=0.0
>> 2015-11-11 00:06:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=569, evicted=0, evictedPerRun=0.0
>> 2015-11-11 00:11:01,089 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=1.63 MB,
>> freeSize=1.55 GB, max=1.55 GB, blockCount=0, accesses=0, hits=0,
>> hitRatio=0, cachingAccesses=0, cachingHits=0,
>> cachingHitsRatio=0,evictions=599, evicted=0, evictedPerRun=0.0
>>
>> Region server logs
>> tail -100
>> hbase-cmf-hbase-REGIONSERVER-hadoop4-private.wdc01.infra.ripple.com.log.out
>> 2015-11-10 22:31:16,255 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,255 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row prod_control,,1434077692105.a4f70c6c01f31f6d3d65e0cf55200c80.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,256 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_agg_account_payments,20140325000000|rH48e8bf5kNWuFRBwgm5sCH133cd9zgV29,1447197624013.c2a9f77791512281e4491e0797e824e8.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,259 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,259 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,259 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,259 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,262 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,262 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,265 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,265 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,265 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,265 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,267 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,268 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,272 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,273 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,274 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> 1fbbd37bdff125be32d352c98dffd2b1; next sequenceid=180083
>> 2015-11-10 22:31:16,274 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> 7ce823a4e0f56bf8c854175409bb17b6; next sequenceid=21535
>> 2015-11-10 22:31:16,275 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for
>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
>> 2015-11-10 22:31:16,276 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for
>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_account_exchanges,rJoom9YSbpxnYcQ47SZzdoDE46iPxtZUA7|20150914171710|000015877580|00023|00001,1447192711873.1fbbd37bdff125be32d352c98dffd2b1.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,278 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_lu_ledgers_by_index,000008965220|F7AC8FF5FCD18033E25CD0D62753E7B7C7FECEDF5DCDE15B8B639293BA770073,1447197130834.7ce823a4e0f56bf8c854175409bb17b6.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,282 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,282 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,282 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> 15701b8131ca505f6cce148d49cfbeee; next sequenceid=1029
>> 2015-11-10 22:31:16,283 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,283 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,284 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for
>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
>> 2015-11-10 22:31:16,304 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_agg_account_stats,day|20150728000000|r3eEo6qg9NDCfE6nZEkX17HweiuFQBoR86,1447197967001.15701b8131ca505f6cce148d49cfbeee.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,308 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,308 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,308 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:16,309 INFO
>> org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
>> size [268435456, 9223372036854775807); files [3, 10); ratio 1.200000;
>> off-peak ratio 5.000000; throttle point 5368709120; major period 604800000,
>> major jitter 0.500000, min locality to compact 0.000000
>> 2015-11-10 22:31:16,318 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> 21b24baeaf26e12d02f79f243076850a; next sequenceid=114628
>> 2015-11-10 22:31:16,318 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Onlined
>> cc52e0ba23708df10d696d0e33d9d137; next sequenceid=161394
>> 2015-11-10 22:31:16,320 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for
>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
>> 2015-11-10 22:31:16,320 INFO
>> org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
>> for prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|2014111212
>> ,1447198690089.cc52e0ba23708df10d696d0e33d9d137.
>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row prod_lu_account_memos,rheYUR2Gz5DAmtbrM7wJPMEhsesiZEZxPD|
>> 2014111212,1447198690089.cc52e0ba23708df10d696d0e33d9d137. with server=
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:16,323 INFO org.apache.hadoop.hbase.MetaTableAccessor:
>> Updated row
>> prod_lu_account_memos,rB5BFH9mQmKJcjkqWt9suhYFVv2mjMXxLj|20151020104,1447193666291.21b24baeaf26e12d02f79f243076850a.
>> with server=hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115
>> 2015-11-10 22:31:21,184 WARN org.apache.hadoop.metrics2.lib.Interns:
>> Metrics intern cache overflow at 2011 for
>> RegionServer={RegionServer=MetricsInfoImpl{name=RegionServer,
>> description=RegionServer}, RegionServer
>> record=MetricsInfoImpl{name=RegionServer, description=RegionServer record}}
>> 2015-11-10 22:31:22,377 WARN
>> org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil:
>> hbase.regionserver.global.memstore.upperLimit is deprecated by
>> hbase.regionserver.global.memstore.size
>> 2015-11-10 22:31:22,403 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Completed compaction of 3 (all) file(s) in f of
>> prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.
>> into d98861e6b5364d819d11c3f3e1c8fb56(size=180.4 M), total size for store
>> is 180.4 M. This selection was in queue for 0sec, and took 7sec to execute.
>> 2015-11-10 22:31:22,405 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>> compaction: Request =
>> regionName=prod_transactions,1FC17DC120A78A242D27D54529C7B09475AE2A33E254236A437DFE78BB3FD8E0,1434468776581.fb145e7787ffc01a8c90dfd48e5b1041.,
>> storeName=f, fileCount=3, fileSize=180.4 M, priority=27,
>> time=741524299592168; duration=7sec
>> 2015-11-10 22:31:22,413 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in
>> region
>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>> 2015-11-10 22:31:22,413 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Starting compaction of 3 file(s) in f of
>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>> into tmpdir=hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_transactions/72f5a6c7dba1e486eae6b75f4e9f0d5d/.tmp,
>> totalSize=82.2 M
>> 2015-11-10 22:31:22,416 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:25,301 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Completed compaction of 3 (all) file(s) in f of
>> prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.
>> into f7678de8dd8149428938a0aedfcbcaf9(size=82.2 M), total size for store is
>> 82.2 M. This selection was in queue for 0sec, and took 2sec to execute.
>> 2015-11-10 22:31:25,302 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>> compaction: Request =
>> regionName=prod_transactions,749C3FD1CAA0AED92E88E27380225CA5D858C41C18A596DCCCD736617E3A511F,1436996811066.72f5a6c7dba1e486eae6b75f4e9f0d5d.,
>> storeName=f, fileCount=3, fileSize=82.2 M, priority=27,
>> time=741532147583369; duration=2sec
>> 2015-11-10 22:31:25,305 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on f in
>> region
>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>> 2015-11-10 22:31:25,305 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Starting compaction of 3 file(s) in f of
>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>> into tmpdir=hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_offers/05284b692219df79852f3deaa2ce1927/.tmp,
>> totalSize=218.4 M
>> 2015-11-10 22:31:25,310 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:39,481 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Completed compaction of 3 (all) file(s) in f of
>> prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.
>> into 4aea6e4301f848ff9bf727f4e511978a(size=214.6 M), total size for store
>> is 214.6 M. This selection was in queue for 0sec, and took 14sec to execute.
>> 2015-11-10 22:31:39,481 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>> compaction: Request =
>> regionName=prod_account_offers,rwBYyfufTzk77zUSKEu4MvixfarC35av1J|20150207013510|000011564024|00019|00001,1447207173407.05284b692219df79852f3deaa2ce1927.,
>> storeName=f, fileCount=3, fileSize=218.4 M, priority=27,
>> time=741535039619615; duration=14sec
>> 2015-11-10 22:31:39,484 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on d in
>> region
>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>> 2015-11-10 22:31:39,484 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Starting compaction of 3 file(s) in d of
>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>> into tmpdir=hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/data/default/prod_account_balance_changes/40d39088cc3efe2d230a7d6f8d7b205b/.tmp,
>> totalSize=197.7 M
>> 2015-11-10 22:31:39,490 INFO
>> org.apache.hadoop.hbase.io.hfile.CacheConfig:
>> blockCache=LruBlockCache{blockCount=0, currentSize=3460104, freeSize=
>> 3369704184, maxSize=3373164288, heapSize=3460104, minSize=3204506112,
>> minFactor=0.95, multiSize=1602253056, multiFactor=0.5,
>> singleSize=801126528, singleFactor=0.25}, cacheDataOnRead=false,
>> cacheDataOnWrite=false, cacheIndexesOnWrite=false,
>> cacheBloomsOnWrite=false, cacheEvictOnClose=false,
>> cacheDataCompressed=false, prefetchOnOpen=false
>> 2015-11-10 22:31:52,393 INFO org.apache.hadoop.hbase.regionserver.HStore:
>> Completed compaction of 3 (all) file(s) in d of
>> prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.
>> into 1e48a57cdb544e898e78c9e8276ec05b(size=197.7 M), total size for store
>> is 197.7 M. This selection was in queue for 0sec, and took 12sec to execute.
>> 2015-11-10 22:31:52,393 INFO
>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed
>> compaction: Request =
>> regionName=prod_account_balance_changes,rDo5Qk2aQPxECHv7ym8wiQiw3YqPbqs2wv|20150123053820|000011281793|00000|00000,1447198896065.40d39088cc3efe2d230a7d6f8d7b205b.,
>> storeName=d, fileCount=3, fileSize=197.7 M, priority=27,
>> time=741549219351203; duration=12sec
>> 2015-11-10 22:32:01,402 INFO
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Atomically
>> moving hadoop3-private.wdc01.infra.ripple.com,60020,1447102836904's wals
>> to my queue
>> 2015-11-10 22:32:01,405 WARN
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer 1 didn't
>> exist, skipping the replay
>> 2015-11-10 22:32:01,406 WARN
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Peer
>> 1-hadoop4-private.wdc01.infra.ripple.com,60020,1447102248964 didn't
>> exist, skipping the replay
>> 2015-11-10 22:32:01,412 WARN
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl: Got exception
>> in copyQueuesFromRSUsingMulti:
>> org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode =
>> Directory not empty
>>     at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:125)
>>     at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1152)
>>     at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:1118)
>>     at
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:685)
>>     at
>> org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1550)
>>     at
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.copyQueuesFromRSUsingMulti(ReplicationQueuesZKImpl.java:340)
>>     at
>> org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.claimQueues(ReplicationQueuesZKImpl.java:172)
>>     at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:570)
>>     at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>     at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>     at java.lang.Thread.run(Thread.java:745)
>> 2015-11-10 22:36:01,223 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=3.78 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=4, accesses=63330, hits=6,
>> hitRatio=0.01%, , cachingAccesses=21, cachingHits=4,
>> cachingHitsRatio=19.05%, evictions=29, evicted=3,
>> evictedPerRun=0.1034482792019844
>> 2015-11-10 22:41:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.15 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=7, accesses=63856, hits=8,
>> hitRatio=0.01%, , cachingAccesses=32, cachingHits=6,
>> cachingHitsRatio=18.75%, evictions=59, evicted=3,
>> evictedPerRun=0.050847455859184265
>> 2015-11-10 22:46:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>> cachingHitsRatio=21.43%, evictions=89, evicted=3,
>> evictedPerRun=0.033707864582538605
>> 2015-11-10 22:51:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>> cachingHitsRatio=21.43%, evictions=119, evicted=3,
>> evictedPerRun=0.02521008439362049
>> 2015-11-10 22:56:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.40 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=9, accesses=63866, hits=11,
>> hitRatio=0.02%, , cachingAccesses=42, cachingHits=9,
>> cachingHitsRatio=21.43%, evictions=149, evicted=3,
>> evictedPerRun=0.020134227350354195
>> 2015-11-10 23:01:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63872, hits=13,
>> hitRatio=0.02%, , cachingAccesses=48, cachingHits=11,
>> cachingHitsRatio=22.92%, evictions=179, evicted=3,
>> evictedPerRun=0.016759777441620827
>> 2015-11-10 23:06:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=63890, hits=21,
>> hitRatio=0.03%, , cachingAccesses=66, cachingHits=19,
>> cachingHitsRatio=28.79%, evictions=209, evicted=3,
>> evictedPerRun=0.014354066923260689
>> 2015-11-10 23:11:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64416, hits=26,
>> hitRatio=0.04%, , cachingAccesses=77, cachingHits=24,
>> cachingHitsRatio=31.17%, evictions=239, evicted=3,
>> evictedPerRun=0.012552301399409771
>> 2015-11-10 23:16:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>> cachingHitsRatio=33.33%, evictions=269, evicted=3,
>> evictedPerRun=0.011152416467666626
>> 2015-11-10 23:21:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>> cachingHitsRatio=33.33%, evictions=299, evicted=3,
>> evictedPerRun=0.010033444501459599
>> 2015-11-10 23:26:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64426, hits=31,
>> hitRatio=0.05%, , cachingAccesses=87, cachingHits=29,
>> cachingHitsRatio=33.33%, evictions=329, evicted=3,
>> evictedPerRun=0.009118541143834591
>> 2015-11-10 23:31:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64432, hits=34,
>> hitRatio=0.05%, , cachingAccesses=93, cachingHits=32,
>> cachingHitsRatio=34.41%, evictions=359, evicted=3,
>> evictedPerRun=0.008356546051800251
>> 2015-11-10 23:31:01,228 INFO org.apache.hadoop.hbase.mob.MobFileCache:
>> MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted
>> files: 0
>> 2015-11-10 23:31:06,790 INFO
>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Rolled WAL /hbase/WALs/
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
>> hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>> with entries=247, filesize=125.29 KB; new WAL /hbase/WALs/
>> hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/
>> hadoop4-private.wdc01.infra.ripple.com
>> %2C60020%2C1447216261115.null0.1447219866745
>> 2015-11-10 23:31:06,792 INFO
>> org.apache.hadoop.hbase.regionserver.wal.FSHLog: Archiving hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/WALs/hadoop4-private.wdc01.infra.ripple.com,60020,1447216261115/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>> to hdfs://
>> hadoop2-private.wdc01.infra.ripple.com:8020/hbase/oldWALs/hadoop4-private.wdc01.infra.ripple.com%2C60020%2C1447216261115.null0.1447216266503
>> 2015-11-10 23:36:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64450, hits=42,
>> hitRatio=0.07%, , cachingAccesses=111, cachingHits=40,
>> cachingHitsRatio=36.04%, evictions=389, evicted=3,
>> evictedPerRun=0.007712082471698523
>> 2015-11-10 23:41:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64976, hits=47,
>> hitRatio=0.07%, , cachingAccesses=122, cachingHits=45,
>> cachingHitsRatio=36.89%, evictions=419, evicted=3,
>> evictedPerRun=0.0071599045768380165
>> 2015-11-10 23:46:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>> cachingHitsRatio=37.88%, evictions=449, evicted=3,
>> evictedPerRun=0.00668151443824172
>> 2015-11-10 23:51:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>> cachingHitsRatio=37.88%, evictions=479, evicted=3,
>> evictedPerRun=0.00626304792240262
>> 2015-11-10 23:56:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64986, hits=52,
>> hitRatio=0.08%, , cachingAccesses=132, cachingHits=50,
>> cachingHitsRatio=37.88%, evictions=509, evicted=3,
>> evictedPerRun=0.005893909838050604
>> 2015-11-11 00:01:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=64992, hits=55,
>> hitRatio=0.08%, , cachingAccesses=138, cachingHits=53,
>> cachingHitsRatio=38.41%, evictions=539, evicted=3,
>> evictedPerRun=0.005565862637013197
>> 2015-11-11 00:06:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65010, hits=63,
>> hitRatio=0.10%, , cachingAccesses=156, cachingHits=61,
>> cachingHitsRatio=39.10%, evictions=569, evicted=3,
>> evictedPerRun=0.005272407550364733
>> 2015-11-11 00:11:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65536, hits=68,
>> hitRatio=0.10%, , cachingAccesses=167, cachingHits=66,
>> cachingHitsRatio=39.52%, evictions=599, evicted=3,
>> evictedPerRun=0.00500834733247757
>> 2015-11-11 00:16:01,222 INFO
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=4.53 MB,
>> freeSize=3.14 GB, max=3.14 GB, blockCount=10, accesses=65546, hits=73,
>> hitRatio=0.11%, , cachingAccesses=177, cachingHits=71,
>> cachingHitsRatio=40.11%, evictions=629, evicted=3,
>> evictedPerRun=0.004769475199282169
>>
>>
>>
>>
>> On Tue, Nov 10, 2015 at 8:15 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>
>>> bq. The region exits  are forcing the data to start transferring to
>>> other regions,
>>>
>>> Can you elaborate a bit more on the above ? Did you mean some regions
>>> were transferred to other region server(s) ?
>>>
>>> Can you determined the effective RegionSplitPolicy ?
>>>
>>> Snippets of server logs would help illustrate the situation.
>>>
>>> Thanks
>>>
>>> On Tue, Nov 10, 2015 at 8:09 PM, Abraham Tom <work2much@gmail.com>
>>> wrote:
>>>
>>>> I am on CDH 5.4.8 hbase 1.0
>>>>
>>>> I am migrating to a new cluster and the data is sourced from CDH 5.3
>>>> hbase
>>>> 0.98.6
>>>>
>>>> I took snapshots from the old cluster exported them to the new cluster
>>>> - no
>>>> errors
>>>>
>>>> cloned the table, flush, major_compact and balanced the regions
>>>>
>>>> the configs are practically identical to the old cluster
>>>>
>>>>
>>>>
>>>> hbck and fsck all check out with no inconsistencies found
>>>>
>>>>
>>>>
>>>> yet when we try to load data via thrift the regions start splitting at
>>>> an
>>>> outrageous pace.
>>>>
>>>> Eventually the regions become offline starting a snowball affect.  The
>>>> region exits  are forcing the data to start transferring to other
>>>> regions,
>>>> with that creates large GC activity and multiple regions in transition
>>>> before thrift can no longer execute any writes.  This usually happens
>>>> after
>>>> a few records.   How we put is that we perform rowkey scans before puts
>>>> to
>>>> ensure where we last left off.
>>>>
>>>>
>>>>
>>>> Thrift eventually stops writing due to regions being offline and no
>>>> longer
>>>> serving.  The regions eventually stop splitting and all the GC
>>>> eventually
>>>> complete.  Once things settle back down, I am able to perform hbck and
>>>> fsck
>>>> and it again comes back with no inconsistencies
>>>>
>>>>
>>>>
>>>> I need ideas as to what is causing this
>>>>
>>>>
>>>> --
>>>> Abraham Tom
>>>> Email:   work2much@gmail.com
>>>> Phone:  415-515-3621
>>>>
>>>
>>>
>>
>>
>> --
>> Abraham Tom
>> Email:   work2much@gmail.com
>> Phone:  415-515-3621
>>
>
>


-- 
Abraham Tom
Email:   work2much@gmail.com
Phone:  415-515-3621

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message