Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 37D8D1891E for ; Wed, 16 Sep 2015 01:02:46 +0000 (UTC) Received: (qmail 90383 invoked by uid 500); 16 Sep 2015 01:02:46 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 90334 invoked by uid 500); 16 Sep 2015 01:02:46 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 90314 invoked by uid 99); 16 Sep 2015 01:02:46 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Sep 2015 01:02:45 +0000 Date: Wed, 16 Sep 2015 01:02:45 +0000 (UTC) From: "Nick Dimiduk (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-12911) Client-side metrics MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-12911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14746612#comment-14746612 ] Nick Dimiduk commented on HBASE-12911: -------------------------------------- That's true, we are looking at "quite a few". On master: {noformat} --- deps.master.txt 2015-09-15 17:47:43.000000000 -0700 +++ deps.12911.txt 2015-09-15 17:47:54.000000000 -0700 @@ -7,13 +7,24 @@ [INFO] --- maven-dependency-plugin:2.8:list (default-cli) @ hbase-client --- [INFO] [INFO] The following files have been resolved: +[INFO] aopalliance:aopalliance:jar:1.0:compile +[INFO] asm:asm:jar:3.1:compile [INFO] com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile [INFO] com.google.code.findbugs:jsr305:jar:1.3.9:compile [INFO] com.google.code.gson:gson:jar:2.2.4:compile [INFO] com.google.guava:guava:jar:12.0.1:compile +[INFO] com.google.inject.extensions:guice-servlet:jar:3.0:compile +[INFO] com.google.inject:guice:jar:3.0:compile [INFO] com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] com.jcraft:jsch:jar:0.1.42:compile +[INFO] com.sun.jersey.contribs:jersey-guice:jar:1.9:compile +[INFO] com.sun.jersey:jersey-client:jar:1.9:compile +[INFO] com.sun.jersey:jersey-core:jar:1.9:compile +[INFO] com.sun.jersey:jersey-json:jar:1.9:compile +[INFO] com.sun.jersey:jersey-server:jar:1.9:compile +[INFO] com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile [INFO] com.thoughtworks.paranamer:paranamer:jar:2.3:compile +[INFO] com.yammer.metrics:metrics-core:jar:2.2.0:compile [INFO] commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] commons-cli:commons-cli:jar:1.2:compile @@ -27,12 +38,17 @@ [INFO] commons-logging:commons-logging:jar:1.2:compile [INFO] commons-net:commons-net:jar:3.1:compile [INFO] io.netty:netty-all:jar:4.0.30.Final:compile +[INFO] javax.activation:activation:jar:1.1:compile +[INFO] javax.inject:javax.inject:jar:1:compile +[INFO] javax.servlet:servlet-api:jar:2.5:compile +[INFO] javax.xml.bind:jaxb-api:jar:2.2.2:compile [INFO] jdk.tools:jdk.tools:jar:1.7:system [INFO] junit:junit:jar:4.12:compile [INFO] log4j:log4j:jar:1.2.17:test [INFO] org.apache.avro:avro:jar:1.7.4:compile [INFO] org.apache.commons:commons-compress:jar:1.4.1:compile [INFO] org.apache.commons:commons-math3:jar:3.1.1:compile +[INFO] org.apache.commons:commons-math:jar:2.2:compile [INFO] org.apache.curator:curator-client:jar:2.7.1:compile [INFO] org.apache.curator:curator-framework:jar:2.7.1:compile [INFO] org.apache.curator:curator-recipes:jar:2.7.1:compile @@ -43,17 +59,27 @@ [INFO] org.apache.hadoop:hadoop-annotations:jar:2.7.1:compile [INFO] org.apache.hadoop:hadoop-auth:jar:2.7.1:compile [INFO] org.apache.hadoop:hadoop-common:jar:2.7.1:compile +[INFO] org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.1:compile +[INFO] org.apache.hadoop:hadoop-yarn-api:jar:2.7.1:compile +[INFO] org.apache.hadoop:hadoop-yarn-common:jar:2.7.1:compile [INFO] org.apache.hbase:hbase-annotations:jar:2.0.0-SNAPSHOT:compile [INFO] org.apache.hbase:hbase-annotations:test-jar:tests:2.0.0-SNAPSHOT:test [INFO] org.apache.hbase:hbase-common:jar:2.0.0-SNAPSHOT:compile [INFO] org.apache.hbase:hbase-common:test-jar:tests:2.0.0-SNAPSHOT:test +[INFO] org.apache.hbase:hbase-hadoop-compat:jar:2.0.0-SNAPSHOT:compile +[INFO] org.apache.hbase:hbase-hadoop-compat:test-jar:tests:2.0.0-SNAPSHOT:test +[INFO] org.apache.hbase:hbase-hadoop2-compat:jar:2.0.0-SNAPSHOT:compile +[INFO] org.apache.hbase:hbase-hadoop2-compat:test-jar:tests:2.0.0-SNAPSHOT:test [INFO] org.apache.hbase:hbase-protocol:jar:2.0.0-SNAPSHOT:compile [INFO] org.apache.htrace:htrace-core:jar:3.1.0-incubating:compile [INFO] org.apache.httpcomponents:httpclient:jar:4.2.5:compile [INFO] org.apache.httpcomponents:httpcore:jar:4.2.4:compile [INFO] org.apache.zookeeper:zookeeper:jar:3.4.6:compile [INFO] org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile +[INFO] org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile [INFO] org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile +[INFO] org.codehaus.jackson:jackson-xc:jar:1.9.13:compile +[INFO] org.codehaus.jettison:jettison:jar:1.3.3:compile [INFO] org.hamcrest:hamcrest-core:jar:1.3:test [INFO] org.hamcrest:hamcrest-library:jar:1.1:test [INFO] org.jmock:jmock-junit4:jar:2.6.0:test {noformat} and on branch-1 backport: {noformat} @@ -13,6 +13,7 @@ [INFO] com.google.protobuf:protobuf-java:jar:2.5.0:compile [INFO] com.jcraft:jsch:jar:0.1.42:compile [INFO] com.thoughtworks.paranamer:paranamer:jar:2.3:compile +[INFO] com.yammer.metrics:metrics-core:jar:2.2.0:compile [INFO] commons-beanutils:commons-beanutils-core:jar:1.8.0:compile [INFO] commons-beanutils:commons-beanutils:jar:1.7.0:compile [INFO] commons-cli:commons-cli:jar:1.2:compile @@ -35,6 +36,7 @@ [INFO] org.apache.avro:avro:jar:1.7.4:compile [INFO] org.apache.commons:commons-compress:jar:1.4.1:compile [INFO] org.apache.commons:commons-math3:jar:3.1.1:compile +[INFO] org.apache.commons:commons-math:jar:2.2:compile [INFO] org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile [INFO] org.apache.directory.api:api-util:jar:1.0.0-M20:compile [INFO] org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile @@ -49,6 +51,10 @@ [INFO] org.apache.hbase:hbase-annotations:test-jar:tests:1.3.0-SNAPSHOT:test [INFO] org.apache.hbase:hbase-common:jar:1.3.0-SNAPSHOT:compile [INFO] org.apache.hbase:hbase-common:test-jar:tests:1.3.0-SNAPSHOT:test +[INFO] org.apache.hbase:hbase-hadoop-compat:jar:1.3.0-SNAPSHOT:compile +[INFO] org.apache.hbase:hbase-hadoop-compat:test-jar:tests:1.3.0-SNAPSHOT:test +[INFO] org.apache.hbase:hbase-hadoop2-compat:jar:1.3.0-SNAPSHOT:compile +[INFO] org.apache.hbase:hbase-hadoop2-compat:test-jar:tests:1.3.0-SNAPSHOT:test [INFO] org.apache.hbase:hbase-protocol:jar:1.3.0-SNAPSHOT:compile [INFO] org.apache.htrace:htrace-core:jar:3.1.0-incubating:compile [INFO] org.apache.httpcomponents:httpclient:jar:4.2.5:compile {noformat} Hmm, maybe I missed something in the backport? Or did someone do a deps purge on master? > Client-side metrics > ------------------- > > Key: HBASE-12911 > URL: https://issues.apache.org/jira/browse/HBASE-12911 > Project: HBase > Issue Type: New Feature > Components: Client, Operability, Performance > Reporter: Nick Dimiduk > Assignee: Nick Dimiduk > Fix For: 2.0.0, 1.3.0 > > Attachments: 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 0001-HBASE-12911-Client-side-metrics.patch, 12911-0.98.00.patch, 12911-branch-1.00.patch, am.jpg, client metrics RS-Master.jpg, client metrics client.jpg, conn_agg.jpg, connection attributes.jpg, ltt.jpg, standalone.jpg > > > There's very little visibility into the hbase client. Folks who care to add some kind of metrics collection end up wrapping Table method invocations with {{System.currentTimeMillis()}}. For a crude example of this, have a look at what I did in {{PerformanceEvaluation}} for exposing requests latencies up to {{IntegrationTestRegionReplicaPerf}}. The client is quite complex, there's a lot going on under the hood that is impossible to see right now without a profiler. Being a crucial part of the performance of this distributed system, we should have deeper visibility into the client's function. > I'm not sure that wiring into the hadoop metrics system is the right choice because the client is often embedded as a library in a user's application. We should have integration with our metrics tools so that, i.e., a client embedded in a coprocessor can report metrics through the usual RS channels, or a client used in a MR job can do the same. > I would propose an interface-based system with pluggable implementations. Out of the box we'd include a hadoop-metrics implementation and one other, possibly [dropwizard/metrics|https://github.com/dropwizard/metrics]. > Thoughts? -- This message was sent by Atlassian JIRA (v6.3.4#6332)