hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nick Dimiduk (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-12911) Client-side metrics
Date Sun, 27 Sep 2015 07:32:05 GMT

     [ https://issues.apache.org/jira/browse/HBASE-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nick Dimiduk updated HBASE-12911:
---------------------------------
    Attachment: 12911.yammer.jpg
                12911.yammer.v00.patch

Here's a rough patch using yammer metrics (our current dependency) instead of hadoop metrics.
Also, a screen shot of the result. Everything is self-contained in the MetricsConnection class
this time, so it's a little easier to see what's happening. I'm not sure how to add the host-based
groups like I did before, so that's not there.

The reporter is hard-coded right now to be JMX, I can look into writing a logger-reporter,
I didn't see one in the library. Also, these metrics will not be exposed over http like our
hadoop metrics currently are. If we want them included in the /jmx endpoint, we'll need to
hack that into our servlet.

As for dependency bloat, this diff is much better.

{noformat}
--- hbase-client/dependencies.master.txt        2015-09-27 09:25:39.000000000 +0200
+++ hbase-client/dependencies.yammer.txt        2015-09-27 09:24:59.000000000 +0200
@@ -14,6 +14,7 @@
 [INFO]    com.google.protobuf:protobuf-java:jar:2.5.0:compile
 [INFO]    com.jcraft:jsch:jar:0.1.42:compile
 [INFO]    com.thoughtworks.paranamer:paranamer:jar:2.3:compile
+[INFO]    com.yammer.metrics:metrics-core:jar:2.2.0:compile
 [INFO]    commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
 [INFO]    commons-beanutils:commons-beanutils:jar:1.7.0:compile
 [INFO]    commons-cli:commons-cli:jar:1.2:compile
{noformat}

> Client-side metrics
> -------------------
>
>                 Key: HBASE-12911
>                 URL: https://issues.apache.org/jira/browse/HBASE-12911
>             Project: HBase
>          Issue Type: New Feature
>          Components: Client, Operability, Performance
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>             Fix For: 2.0.0, 1.3.0
>
>         Attachments: 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch,
0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch,
0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch,
0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch,
12911-0.98.00.patch, 12911-branch-1.00.patch, 12911.yammer.jpg, 12911.yammer.v00.patch, am.jpg,
client metrics RS-Master.jpg, client metrics client.jpg, conn_agg.jpg, connection attributes.jpg,
ltt.jpg, standalone.jpg
>
>
> There's very little visibility into the hbase client. Folks who care to add some kind
of metrics collection end up wrapping Table method invocations with {{System.currentTimeMillis()}}.
For a crude example of this, have a look at what I did in {{PerformanceEvaluation}} for exposing
requests latencies up to {{IntegrationTestRegionReplicaPerf}}. The client is quite complex,
there's a lot going on under the hood that is impossible to see right now without a profiler.
Being a crucial part of the performance of this distributed system, we should have deeper
visibility into the client's function.
> I'm not sure that wiring into the hadoop metrics system is the right choice because the
client is often embedded as a library in a user's application. We should have integration
with our metrics tools so that, i.e., a client embedded in a coprocessor can report metrics
through the usual RS channels, or a client used in a MR job can do the same.
> I would propose an interface-based system with pluggable implementations. Out of the
box we'd include a hadoop-metrics implementation and one other, possibly [dropwizard/metrics|https://github.com/dropwizard/metrics].
> Thoughts?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message