hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samir Ahmic <ahmic.sa...@gmail.com>
Subject Re: Very High CPU usage in RegionServer
Date Fri, 08 Jul 2016 06:57:59 GMT
Hi Sandeep,

What sort of load is on cluster when this is happening ? What logs say in
moment when cluster is this state ? It is strange that whole cluster
is unresponsive i can remember few cases when this is possible:

   - "hbase:meta" table is in transition or unavailable
   - zookeeper is overloaded with huge number of connections
   - HDFS isues

Check logs there must be explanation why you have such huge resource usage
spikes.

Regards
Samir

On Fri, Jul 8, 2016 at 8:06 AM, Sandeep Reddy <sandeepvreddy@outlook.com>
wrote:

> Hi,
>
>
> We are observing very high CPU load(400 to 600%) in one of our
> RegionServer to the point where the machine is becoming unresponsive.
>
> At this point the whole cluster of 20+ RegionServers becoming unresponsive.
>
> Before cluster becomes unresponsive we observed following symptoms:
>
>   *   Huge bandwidth spike
>   *   CPU spikes vertically form normal load to very high usage only in
> one RegionServer
>   *   Few times even though machine is unresponsive, it sending heartbeats
> to master
>   *   There is no spike in number of requests to HBase
>   *   We are observed this pattern at least twice is last week
>   *   We don't have any co-processors in any of the region servers
>
> What could be the possible reasons for this kind of behaviour.
>
> We are using hbase-0.98.7, hadoop-2.5.1 versions.
> Its production cluster so upgrading to latest version will not be possible
> right away.
>
>
> Thanks,
> Sandeep.
>
>
>
> Thanks,
> Sandeep.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message