hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: Region Server Thread with a Single High Idle CPU
Date Fri, 19 Dec 2014 19:34:57 GMT
Hi Jon,

Do you see something interesting in the RS logs from KVM15 or the HBase
Master? one possibility is that if there are no requests to META coming
from the Thrift server or external clients, then it might be possible that
one or many region servers for some reason are updating META too frequently
(e.g. 1000s of regions in transition or a region that might be corrupt
region and is getting resubmitted over and over by the MetaEditor for some
kind of failure).  If you could  enable DEBUG level logging in the RSs and
the Master for a short period when you see that large number of requests to
META that can help us to find out why is that happening.

cheers,
esteban.


--
Cloudera, Inc.


On Fri, Dec 19, 2014 at 9:09 AM, uamadman <uamadman1@gmail.com> wrote:
>
> Yes,
> I tested the following by restarting the cluster and waiting approximately
> 5-10 minutes for its initial ramp up. There are no clients asking for data.
> In the following example KVM15 was randomly assigned to serve the META
> Table.
>
> root@KVM15:~# lsof -n | grep :60020- | sed 's/.*->//;s/:.*//' | sort |
> uniq
> -c | sort -nr
>       1 10.40.178.180
>       1 10.40.178.179
>       1 10.40.178.176
>       1 10.40.178.170
>       1 10.40.178.169
>       1 10.40.178.164
>       1 10.40.178.162
>       1 10.40.178.160
>       1 10.40.178.130
>       1 10.40.178.129
>       1 10.40.178.128
>       1 10.40.178.110
>       1 10.40.178.109
>       1 10.40.178.108
>
> The following returned nothing
> root@KVM15:~# lsof -n | grep :9090- | sed 's/.*->//;s/:.*//' | sort | uniq
> -c | sort -nr
>
>
> This picture shows the iftop  for KVM15 LeftSide | KVM03 Top Right
>                                                                    | KVM05
> Bottom Right
> In this example it was running for about 1-2 minutes before i took the
> screen shot and had already transmitted ~2GB of data.
> https://www.dropbox.com/s/iy0rkgljyo1872m/KVMThroughputExample.png?dl=0
>
> At the 10 minute mark KVM15 had transmitted ~18GB of data.
> https://www.dropbox.com/s/cgpo1krebrp2jzh/KVMThroughputExample10.png?dl=0
>
> Cheers,
> Jon
>
> PS. I'm currently interested in moving from 5.1.2-1.cdh5.1.2.p0.3 to
> 5.2.1-1.cdh5.2.1.p0.12
> and from HBASE 0.98.1 to HBASE 0.98.6. But as im in no rush, I will wait
> until you guys feel you know what this issue is.
>
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Region-Server-Thread-with-a-Single-High-Idle-CPU-tp4066876p4067014.html
> Sent from the HBase User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message