Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 334F6191BF for ; Tue, 22 Mar 2016 08:22:26 +0000 (UTC) Received: (qmail 9276 invoked by uid 500); 22 Mar 2016 08:22:26 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 9221 invoked by uid 500); 22 Mar 2016 08:22:26 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 9175 invoked by uid 99); 22 Mar 2016 08:22:25 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Mar 2016 08:22:25 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id A38512C1F62 for ; Tue, 22 Mar 2016 08:22:25 +0000 (UTC) Date: Tue, 22 Mar 2016 08:22:25 +0000 (UTC) From: "Hudson (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-15464) Flush / Compaction metrics revisited MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-15464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205992#comment-15205992 ] Hudson commented on HBASE-15464: -------------------------------- FAILURE: Integrated in HBase-1.3 #616 (See [https://builds.apache.org/job/HBase-1.3/616/]) HBASE-15464 Flush / Compaction metrics revisited (enis: rev 179cd14ce11e38fb1e32f0ad7fac4b115251612f) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionContext.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMetricsRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreEngine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/flush/FlushTableSubprocedure.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreEngine.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreEngine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionRequest.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlushContext.java > Flush / Compaction metrics revisited > ------------------------------------ > > Key: HBASE-15464 > URL: https://issues.apache.org/jira/browse/HBASE-15464 > Project: HBase > Issue Type: Sub-task > Components: metrics > Reporter: Enis Soztutar > Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: hbase-15464_v1.patch, hbase-15464_v2.patch, hbase-15464_v3.patch, hbase-15464_v3.patch, hbase-15464_v4.patch > > > We can add a couple of metrics related to flushes and compactions: > - flush memstore and output file size histogram: This will allow seeing whether we are flushing too early due to memory pressure, too many regions, etc. Tracking flush memstore size vs output file size is useful in understanding the block encoding compression benefits. > - total flushed output bytes: This will allow to monitor the IO / throughput from flushers. You can use this to set num flushers, flush throttle, etc. > - smallCompactionQueueLength / large...: This is tracked, but not emitted anymore due to a bug. > - compaction time histogram: similar to flush time histogram, how long compactions are taking. > - compaction input num files / output num files histogram: How many files on average we are compacting. Stripe compaction / date tiered compaction can use the num output files metric. > - compaction input / output data sizes histogram: How much data on average we are compacting. > - compaction input / output total bytes: Measure compaction IO / throughput. measure write amplification, enables to set compaction throttle. > - Breakdown for above for major compactions -- This message was sent by Atlassian JIRA (v6.3.4#6332)