hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elliott Clark (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HBASE-10436) hbase 0.96+ jmx does not have regionserver info any more.
Date Wed, 29 Jan 2014 19:28:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885653#comment-13885653
] 

Elliott Clark edited comment on HBASE-10436 at 1/29/14 7:27 PM:
----------------------------------------------------------------

{quote}This is intentional and actually the point. It is pretty clear from the JMX (java management
extensions) documentation [1] that it is a management interface - not just a metrics interface.
It is intended to be a common interface for managing java services. Services could be designed
to accept input and provide information all via the JMX interface and could be designed (if
facilities were provided) to manage and runtime configuration changes.
{quote}

Why would you want some of the metrics to be accessible though the hadoop metrics system and
not others ?  People are using the hadoop sinks to get this information.  I'm -1 on not being
consistent. A ton of work was done to make the metrics regular and all exposed through the
new hadoop metrics2 system. The hadoop metrics system will expose all of this through jmx
and will also expose it to the sinks plugins.  In my opinion if this functionality was really
wanted (and I'm not convinced that it is) it should be as a single source that isn't registered
at all unless a configuration is set.  And we should deprecate exposing anything but master
metrics through the master.

{quote}
Having separate jmx modules allows the user to selectively poll information and thus avoid
any cost associated with gathering the online regionserver list [2]. Furthermore, this is
at the master and I contend that any synchronization would actually be fairly cheap – I
don't believe there is evidence that there is high contention on this online regionserver
list. This only retrieves info when polled - its is not a "NotificationEmmitter". If it was
truly onerous, I would make the same argument that we shouldn't provide this info in the hbase
master webpage and we dare not auto refresh the page (what a NotificationEmitter would do)
because of the synchronization costs.
{quote}


If someone is connected to the JMX port the info is usually polled and streamed (see opentsdb's
implementation of jmx polling).  Meaning that every few seconds the online region servers
will be locked and copy of the collection will be made.  I would contend that having the web
page show it once is very different from somehting that will be know to poll every second.

{quote}
JMX is a management api protocol not just meant to server metrics.
{quote}
HBase has never used it as more than just another way to get metrics.  We shouldn't start
now.  This patch doesn't expose that so I don't understand bringing it up at all.


{quote}
Do external programs plug into this api? Does a user have to change code if the api changes?
yes. 
Should users need to significantly modify their applications when deciding to upgrade? ideally
no.
{quote}

By your definitions the log messages emitted from log4j can't be changed if monitoring system
is tailing the logs.  I think your criteria are way too loose. Monitoring a service by is
less important than using a service and so the metrics and things used by monitoring should
have a lower bar.

{quote}
Regarding the examples – each hlog or memstore implementation would have a different bean
{quote}

But then the unused beans would either need to be removed or worse would emit wrong metrics.
 Hence anyone monitoring the beans will need to change their monitoring code. It's way too
onerous to expect that everything that exposes a metric must always expose that metric.

{quote}
What is your suggestion on how to get this information?
{quote}

Every RegionServer already exposes all of this information.


was (Author: eclark):
{quote}This is intentional and actually the point. It is pretty clear from the JMX (java management
extensions) documentation [1] that it is a management interface - not just a metrics interface.
It is intended to be a common interface for managing java services. Services could be designed
to accept input and provide information all via the JMX interface and could be designed (if
facilities were provided) to manage and runtime configuration changes.
{quote}

Why would you want some of the metrics to be accessible though the hadoop metrics system and
not others ?  People are using the hadoop sinks to get this information.  I'm -1 on not being
consistent. A ton of work was done to make the metrics regular and all exposed through the
new hadoop metrics2 system. The hadoop metrics system will expose all of this through jmx
and will also expose it to the sinks plugins.  In my opinion if this functionality was really
wanted (and I'm not convinced that it is)

{quote}
Having separate jmx modules allows the user to selectively poll information and thus avoid
any cost associated with gathering the online regionserver list [2]. Furthermore, this is
at the master and I contend that any synchronization would actually be fairly cheap – I
don't believe there is evidence that there is high contention on this online regionserver
list. This only retrieves info when polled - its is not a "NotificationEmmitter". If it was
truly onerous, I would make the same argument that we shouldn't provide this info in the hbase
master webpage and we dare not auto refresh the page (what a NotificationEmitter would do)
because of the synchronization costs.
{quote}


If someone is connected to the JMX port the info is usually polled and streamed (see opentsdb's
implementation of jmx polling).  Meaning that every few seconds the online region servers
will be locked and copy of the collection will be made.  I would contend that having the web
page show it once is very different from somehting that will be know to poll every second.

{quote}
JMX is a management api protocol not just meant to server metrics.
{quote}
HBase has never used it as more than just another way to get metrics.  We shouldn't start
now.  This patch doesn't expose that so I don't understand bringing it up at all.


{quote}
Do external programs plug into this api? Does a user have to change code if the api changes?
yes. 
Should users need to significantly modify their applications when deciding to upgrade? ideally
no.
{quote}

By your definitions the log messages emitted from log4j can't be changed if monitoring system
is tailing the logs.  I think your criteria are way too loose. Monitoring a service by is
less important than using a service and so the metrics and things used by monitoring should
have a lower bar.

{quote}
Regarding the examples – each hlog or memstore implementation would have a different bean
{quote}

But then the unused beans would either need to be removed or worse would emit wrong metrics.
 Hence anyone monitoring the beans will need to change their monitoring code. It's way too
onerous to expect that everything that exposes a metric must always expose that metric.

{quote}
What is your suggestion on how to get this information?
{quote}

Every RegionServer already exposes all of this information.

> hbase 0.96+ jmx does not have regionserver info any more.
> ---------------------------------------------------------
>
>                 Key: HBASE-10436
>                 URL: https://issues.apache.org/jira/browse/HBASE-10436
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.98.0, 0.96.0, 0.99.0
>            Reporter: Jonathan Hsieh
>            Assignee: Jonathan Hsieh
>            Priority: Critical
>         Attachments: hbase-10436.patch, hbase-10436.v2.patch
>
>
> HBase 0.96's refactored jmx beans do not contain the master's list of dead region servers
and live regionservers with load info.  HBase 0.94 did (though in a single monolithic blob).
 
> This JMX interface should be considered as much of an API as the the normal wire or java
api.  Dropping values from this was done without deprecation and the removal of this information
is a functional regression.
> We should provide the information in the 0.96+ JMX.  HBase 0.94 had a  monolithic JMX
blob ("hadoop:service=Master,name=Master")  that contained a lot of information, including
the regionserver list and the cached regionserver load for each region  found on the master
webpage.  0.96+ refactored jmx this into several jmx beans which could be selectively retrieved.
 These include:
> * hadoop:service=HBase,name=Master,sub=AssignmentManager
> * hadoop:service=HBase,name=Master,sub=Balancer
> * hadoop:service=HBase,name=Master,sub=Server
> * hadoop:service=HBase,name=Master,sub=FileSystem
> Specifically the (Hadoop:service=HBase,name=Master,sub=Server) listing that used to contain
regionservers and deadregionservers in jmx were replaced in   with numRegionServers and numDeadRegionservers
which only contain counts.  
> I propose just adding another mbean called "RegionServers" under the bean: "hadoop:service=HBase,name=Master,sub=RegionServers"



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message