hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sterfield <sterfi...@gmail.com>
Subject Spikes in JMX metrics gathered on Region Servers
Date Fri, 12 Aug 2016 08:19:33 GMT
Hello,

I have a small issue on Region Servers. I'm gathering all metrics using JMX
but from time to time, I can see some huge spikes in my graphs [1].

At first, I was thinking that the counter had wrapped, but by displaying
the "real" value (not the rate), you can see that two different values has
been sent to OpenTSDB [2] [3]. So when doing the derivative, the result is
a big number, far bigger than the others, hence the spike.

By watching the log at the same time, I can see that the Hbase metrics
system has been stopped then restarted few seconds before. However this
restart of the metric systems happen every 5 minutes on the region servers,
and I don't see spikes in my metric systems so often.

2016-08-11 05:40:08,174 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl:
Stopping HBase metrics system...

Does anyone has some clue on what's happening ?
I'm using HBase 1.2.2.

Thanks,

Guillaume

[1] : https://www.dropbox.com/s/d6v2rm7slvg27kk/JMX%20spike.png?dl=0
[2] :
https://www.dropbox.com/s/ehktdebnzmu2ac7/JMX%20real%20value%201.png?dl=0
[3] :
https://www.dropbox.com/s/75cy4gde8ifsfqy/JMX%20real%20value%202.png?dl=0

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message