Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of ndimiduk@gmail.com designates
 209.85.160.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <A02C490A-84E6-490C-81A5-F5930AB9AAE4@whisper.sh>
References: <E540A926-631A-4FA4-A738-46D5993F220A@whisper.sh>
	<CALte62xtFUeJe-aOiDFqg_7i7A=e6YoZ+XEk-bsUdDo3Uf+Q+g@mail.gmail.com>
	<CALte62yjuS0=ufYNwRK3PM8sa3Fo-hrHK+ahdcQKOGhRckVtHA@mail.gmail.com>
	<89A10F47-55E9-474C-BB20-206D73B4DF53@whisper.sh>
	<CALte62w9jFt1WAeS_nWWeV-8sHiNTRgYg0jGLZJwsUFdrx1Low@mail.gmail.com>
	<EC2B6038-B4FB-43EA-88A6-EE18DB244218@whisper.sh>
	<CALte62yBMC3Cq0gM0_xHCiyDpZ2NKDShvT389-pO7U2qw28wKg@mail.gmail.com>
	<1FACBBCE-5030-4A24-97FA-69F14C3DD7AF@whisper.sh>
	<CALte62xmsMearaW0WT7qNND820o+axMBHCmFK9ptHBpMpxo5tA@mail.gmail.com>
	<68932EEA-CC83-4910-ABA8-72A46E78C3F6@whisper.sh>
	<CANZDn9vMkwRvQoChmHz1NSdtR6+H=evj3iE48u=yK0_+dovfog@mail.gmail.com>
	<CDB764E6-8E4C-4939-8F0B-D312C35375D3@whisper.sh>
	<CANZDn9umqUG1UZ500YEO-+-LbC1D3=gkkp2EsCvWBDqs6-85ig@mail.gmail.com>
	<F56FDC02-6CA5-44D5-BDA0-863554868009@whisper.sh>
	<CANZDn9sQaqfrJN+Zpid+OzrPE0c=qzhzh_Xoc02n-XqX3xDnuA@mail.gmail.com>
	<A02C490A-84E6-490C-81A5-F5930AB9AAE4@whisper.sh>
Date: Thu, 6 Nov 2014 14:09:51 -0800
Message-ID: 
 <CANZa=GuL9bSTFT0KiHN5DwCVMo3Xc7FyjVEhhOso-q6kxgPjvA@mail.gmail.com>
Subject: Re: Hbase Unusable after auto split to 1024 regions
From: Nick Dimiduk <ndimiduk@gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11c2009896eac4050737f417

--001a11c2009896eac4050737f417
Content-Type: text/plain; charset=UTF-8

Ive been doing some testing with ITMTTR recently in ec2 with m1.l and m1.xl
instances. Debug level logging seems to produce 20-30 messages/sec on the
RS. I have noticed pauses in the log entries that last anywhere from 30-120
seconds. I have no explanation for the pauses other than the environment.
(Oddly, the JvmPauseMonitor thread is not detecting these events.) You may
experience similar behavior.

If you're planning production for ec2 with HBase, I do recommend the newer
instance types. Basically: whatever they run Redshift on will be much more
reliable than the stuff they offer to the commoners.

-n

On Thursday, November 6, 2014, Pere Kyle <pere@whisper.sh> wrote:

> Bryan,
>
> Thanks again for the incredibly useful reply.
>
> I have confirmed that the callQueueLen is in fact 0, with a max value of 2
> in the last week (in ganglia)
>
> hbase.hstore.compaction.max was set to 15 on the nodes, from a previous 7.
>
> Freezes (laggy responses) on the cluster are frequent and affect both
> reads and writes. I noticed iowait on the nodes that spikes.
>
> The cluster goes between a state of working 100% to nothing
> serving/timeouts for no discernible reason.
>
> Looking through the logs I have tons of responseTooSlow, this is the only
> regular occurrence in the logs:
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 39
> on 60020): (responseTooSlow):
> {"processingtimems":14573,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@c67b2ac),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.139.198:57223
> ","starttimems":1415246057066,"queuetimems":20640,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 42
> on 60020): (responseTooSlow):
> {"processingtimems":45660,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6c034090),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.21.106:41126
> ","starttimems":1415246025979,"queuetimems":202,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 46
> on 60020): (responseTooSlow):
> {"processingtimems":14620,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4fc3bb1f),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.230.130.102:54068
> ","starttimems":1415246057021,"queuetimems":27565,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 35
> on 60020): (responseTooSlow):
> {"processingtimems":13431,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b321922),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.227.42.252:60493
> ","starttimems":1415246058210,"queuetimems":1134,"class":"HRegionServer","responsesize":0,"method":"multi"}
> On Nov 6, 2014, at 12:38 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
> <javascript:;>> wrote:
>
> > blockingStoreFiles
>
>

--001a11c2009896eac4050737f417--