hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15464) Flush / Compaction metrics revisited
Date Sat, 19 Mar 2016 09:18:33 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202674#comment-15202674
] 

Hadoop QA commented on HBASE-15464:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} |
{color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 5 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s {color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 34s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 40s {color} |
{color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 12s {color} |
{color:green} master passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 13s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 7s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m 39s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 36s {color} |
{color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 45s {color} |
{color:green} master passed with JDK v1.7.0_79 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s {color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 16s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 46s {color} |
{color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 46s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 52s {color} |
{color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 52s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 15s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m 8s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s {color}
| {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 50m 40s {color}
| {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2
2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 12m 55s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 33s {color} |
{color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 41s {color} |
{color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s {color} | {color:green}
hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s {color} | {color:green}
hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s {color} | {color:green}
hbase-hadoop2-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 42s {color} | {color:green}
hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 42s {color} | {color:red}
hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 53s {color} | {color:red}
hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 7m 4s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 343m 15s {color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
|   | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 |
| Timed out junit tests | org.apache.hadoop.hbase.regionserver.TestCorruptedRegionStoreFile
|
|   | org.apache.hadoop.hbase.regionserver.TestSplitLogWorker |
|   | org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy |
|   | org.apache.hadoop.hbase.regionserver.TestCompaction |
|   | org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
|   | org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook |
|   | org.apache.hadoop.hbase.wal.TestWALFiltering |
|   | org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver |
|   | org.apache.hadoop.hbase.wal.TestDefaultWALProviderWithHLogKey |
|   | org.apache.hadoop.hbase.TestZooKeeper |
|   | org.apache.hadoop.hbase.wal.TestWALSplitCompressed |
|   | org.apache.hadoop.hbase.regionserver.TestColumnSeeking |
|   | org.apache.hadoop.hbase.regionserver.TestRegionIncrement |
|   | org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort |
|   | org.apache.hadoop.hbase.regionserver.TestCompactSplitThread |
|   | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
|   | org.apache.hadoop.hbase.util.TestHBaseFsckEncryption |
|   | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
|   | org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss |
|   | org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders |
|   | org.apache.hadoop.hbase.regionserver.TestRegionServerReportForDuty |
|   | org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential |
|   | org.apache.hadoop.hbase.regionserver.TestReversibleScanners |
|   | org.apache.hadoop.hbase.regionserver.wal.TestLogRolling |
|   | org.apache.hadoop.hbase.util.TestFanOutOneBlockAsyncDFSOutput |
|   | org.apache.hadoop.hbase.regionserver.throttle.TestCompactionWithThroughputController
|
|   | org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
|   | org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingTTL |
|   | org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster |
|   | org.apache.hadoop.hbase.regionserver.TestJoinedScanners |
|   | org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
|   | org.apache.hadoop.hbase.mob.TestExpiredMobFileCleaner |
|   | org.apache.hadoop.hbase.TestAcidGuarantees |
|   | org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor |
|   | org.apache.hadoop.hbase.TestGlobalMemStoreSize |
|   | org.apache.hadoop.hbase.io.hfile.TestSeekBeforeWithInlineBlocks |
|   | org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex |
|   | org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks |
|   | org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter |
|   | org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster |
|   | org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock |
|   | org.apache.hadoop.hbase.TestRegionRebalancing |
|   | org.apache.hadoop.hbase.TestMovedRegionsCleaner |
|   | org.apache.hadoop.hbase.regionserver.TestRecoveredEdits |
|   | org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelReplicationWithExpAsString
|
|   | org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsReplication |
|   | org.apache.hadoop.hbase.TestHBaseOnOtherDfsCluster |
|   | org.apache.hadoop.hbase.util.TestMergeTable |
|   | org.apache.hadoop.hbase.util.TestMergeTool |
|   | org.apache.hadoop.hbase.util.TestRegionMover |
|   | org.apache.hadoop.hbase.io.hfile.TestScannerFromBucketCache |
|   | org.apache.hadoop.hbase.regionserver.TestHRegion |
|   | org.apache.hadoop.hbase.regionserver.TestRSKilledWhenInitializing |
|   | org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel |
|   | org.apache.hadoop.hbase.regionserver.TestDefaultMemStore |
|   | org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite |
|   | org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk |
|   | org.apache.hadoop.hbase.TestLocalHBaseCluster |
|   | org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner |
|   | org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed |
|   | org.apache.hadoop.hbase.regionserver.TestRegionServerHostname |
|   | org.apache.hadoop.hbase.regionserver.TestClusterId |
|   | org.apache.hadoop.hbase.regionserver.TestHMobStore |
|   | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
|   | org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents |
|   | org.apache.hadoop.hbase.wal.TestDefaultWALProvider |
|   | org.apache.hadoop.hbase.security.token.TestTokenAuthentication |
|   | org.apache.hadoop.hbase.regionserver.TestSeekOptimizations |
|   | org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort |
|   | org.apache.hadoop.hbase.io.hfile.TestHFileBlock |
|   | org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover |
|   | org.apache.hadoop.hbase.util.TestConnectionCache |
|   | org.apache.hadoop.hbase.regionserver.TestStore |
|   | org.apache.hadoop.hbase.wal.TestWALFactory |
|   | org.apache.hadoop.hbase.wal.TestBoundedRegionGroupingStrategy |
|   | org.apache.hadoop.hbase.regionserver.TestParallelPut |
|   | org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad |
|   | org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat |
|   | org.apache.hadoop.hbase.regionserver.TestAtomicOperation |
|   | org.apache.hadoop.hbase.security.token.TestZKSecretWatcher |
|   | org.apache.hadoop.hbase.io.TestFileLink |
|   | org.apache.hadoop.hbase.io.encoding.TestBufferedDataBlockEncoder |
|   | org.apache.hadoop.hbase.regionserver.TestMajorCompaction |
|   | org.apache.hadoop.hbase.regionserver.TestMinorCompaction |
|   | org.apache.hadoop.hbase.util.TestFSUtils |
|   | org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers |
|   | org.apache.hadoop.hbase.wal.TestWALSplit |
|   | org.apache.hadoop.hbase.regionserver.TestCorruptedRegionStoreFile |
|   | org.apache.hadoop.hbase.regionserver.TestSplitLogWorker |
|   | org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy |
|   | org.apache.hadoop.hbase.regionserver.TestCompaction |
|   | org.apache.hadoop.hbase.snapshot.TestSnapshotClientRetries |
|   | org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook |
|   | org.apache.hadoop.hbase.wal.TestWALFiltering |
|   | org.apache.hadoop.hbase.coprocessor.TestRegionServerObserver |
|   | org.apache.hadoop.hbase.wal.TestDefaultWALProviderWithHLogKey |
|   | org.apache.hadoop.hbase.TestZooKeeper |
|   | org.apache.hadoop.hbase.wal.TestWALSplitCompressed |
|   | org.apache.hadoop.hbase.regionserver.TestColumnSeeking |
|   | org.apache.hadoop.hbase.regionserver.TestRegionIncrement |
|   | org.apache.hadoop.hbase.regionserver.wal.TestLogRollAbort |
|   | org.apache.hadoop.hbase.regionserver.TestCompactSplitThread |
|   | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
|   | org.apache.hadoop.hbase.util.TestHBaseFsckEncryption |
|   | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
|   | org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss |
|   | org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders |
|   | org.apache.hadoop.hbase.regionserver.TestRegionServerReportForDuty |
|   | org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential |
|   | org.apache.hadoop.hbase.regionserver.TestReversibleScanners |
|   | org.apache.hadoop.hbase.regionserver.wal.TestLogRolling |
|   | org.apache.hadoop.hbase.util.TestFanOutOneBlockAsyncDFSOutput |
|   | org.apache.hadoop.hbase.regionserver.throttle.TestCompactionWithThroughputController
|
|   | org.apache.hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
|   | org.apache.hadoop.hbase.io.hfile.TestScannerSelectionUsingTTL |
|   | org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster |
|   | org.apache.hadoop.hbase.regionserver.TestJoinedScanners |
|   | org.apache.hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
|   | org.apache.hadoop.hbase.mob.TestExpiredMobFileCleaner |
|   | org.apache.hadoop.hbase.TestAcidGuarantees |
|   | org.apache.hadoop.hbase.regionserver.TestCompactionWithCoprocessor |
|   | org.apache.hadoop.hbase.TestGlobalMemStoreSize |
|   | org.apache.hadoop.hbase.io.hfile.TestSeekBeforeWithInlineBlocks |
|   | org.apache.hadoop.hbase.io.hfile.TestHFileBlockIndex |
|   | org.apache.hadoop.hbase.io.hfile.TestForceCacheImportantBlocks |
|   | org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter |
|   | org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster |
|   | org.apache.hadoop.hbase.zookeeper.lock.TestZKInterProcessReadWriteLock |
|   | org.apache.hadoop.hbase.TestRegionRebalancing |
|   | org.apache.hadoop.hbase.TestMovedRegionsCleaner |
|   | org.apache.hadoop.hbase.regionserver.TestRecoveredEdits |
|   | org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelReplicationWithExpAsString
|
|   | org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsReplication |
|   | org.apache.hadoop.hbase.TestHBaseOnOtherDfsCluster |
|   | org.apache.hadoop.hbase.util.TestMergeTable |
|   | org.apache.hadoop.hbase.util.TestMergeTool |
|   | org.apache.hadoop.hbase.util.TestRegionMover |
|   | org.apache.hadoop.hbase.io.hfile.TestScannerFromBucketCache |
|   | org.apache.hadoop.hbase.regionserver.TestHRegion |
|   | org.apache.hadoop.hbase.regionserver.TestRSKilledWhenInitializing |
|   | org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel |
|   | org.apache.hadoop.hbase.regionserver.TestDefaultMemStore |
|   | org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite |
|   | org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk |
|   | org.apache.hadoop.hbase.TestLocalHBaseCluster |
|   | org.apache.hadoop.hbase.regionserver.TestMultiColumnScanner |
|   | org.apache.hadoop.hbase.regionserver.TestFSErrorsExposed |
|   | org.apache.hadoop.hbase.regionserver.TestRegionServerHostname |
|   | org.apache.hadoop.hbase.regionserver.TestClusterId |
|   | org.apache.hadoop.hbase.regionserver.TestHMobStore |
|   | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
|   | org.apache.hadoop.hbase.regionserver.TestHRegionReplayEvents |
|   | org.apache.hadoop.hbase.wal.TestDefaultWALProvider |
|   | org.apache.hadoop.hbase.security.token.TestTokenAuthentication |
|   | org.apache.hadoop.hbase.regionserver.TestSeekOptimizations |
|   | org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort |
|   | org.apache.hadoop.hbase.io.hfile.TestHFileBlock |
|   | org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover |
|   | org.apache.hadoop.hbase.util.TestConnectionCache |
|   | org.apache.hadoop.hbase.regionserver.TestStore |
|   | org.apache.hadoop.hbase.wal.TestWALFactory |
|   | org.apache.hadoop.hbase.wal.TestBoundedRegionGroupingStrategy |
|   | org.apache.hadoop.hbase.regionserver.TestParallelPut |
|   | org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoad |
|   | org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat |
|   | org.apache.hadoop.hbase.regionserver.TestAtomicOperation |
|   | org.apache.hadoop.hbase.security.token.TestZKSecretWatcher |
|   | org.apache.hadoop.hbase.io.TestFileLink |
|   | org.apache.hadoop.hbase.io.encoding.TestBufferedDataBlockEncoder |
|   | org.apache.hadoop.hbase.regionserver.TestMajorCompaction |
|   | org.apache.hadoop.hbase.regionserver.TestMinorCompaction |
|   | org.apache.hadoop.hbase.util.TestFSUtils |
|   | org.apache.hadoop.hbase.io.encoding.TestEncodedSeekers |
|   | org.apache.hadoop.hbase.wal.TestWALSplit |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12794292/hbase-15464_v3.patch
|
| JIRA Issue | HBASE-15464 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  hbaseanti  checkstyle
 compile  |
| uname | Linux penates.apache.org 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3a6d683 |
| Default Java | 1.7.0_79 |
| Multi-JDK versions |  /home/jenkins/tools/java/jdk1.8.0:1.8.0 /usr/local/jenkins/java/jdk1.7.0_79:1.7.0_79
|
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/1080/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/1080/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |  https://builds.apache.org/job/PreCommit-HBASE-Build/1080/artifact/patchprocess/patch-unit-hbase-server.txt
https://builds.apache.org/job/PreCommit-HBASE-Build/1080/artifact/patchprocess/patch-unit-hbase-server.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/1080/testReport/ |
| modules | C:  hbase-hadoop-compat   hbase-hadoop-compat   hbase-hadoop2-compat   hbase-hadoop2-compat
  hbase-server   hbase-server  U: . |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/1080/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> Flush / Compaction metrics revisited
> ------------------------------------
>
>                 Key: HBASE-15464
>                 URL: https://issues.apache.org/jira/browse/HBASE-15464
>             Project: HBase
>          Issue Type: Sub-task
>          Components: metrics
>            Reporter: Enis Soztutar
>            Assignee: Enis Soztutar
>             Fix For: 2.0.0, 1.3.0, 1.4.0
>
>         Attachments: hbase-15464_v1.patch, hbase-15464_v2.patch, hbase-15464_v3.patch,
hbase-15464_v3.patch
>
>
> We can add a couple of metrics related to flushes and compactions: 
>  - flush memstore and output file size histogram: This will allow seeing whether we are
flushing too early due to memory pressure, too many regions, etc. Tracking flush memstore
size vs output file size is useful in understanding the block encoding compression benefits.

>  - total flushed output bytes: This will allow to monitor the IO / throughput from flushers.
You can use this to set num flushers, flush throttle, etc. 
>  - smallCompactionQueueLength / large...: This is tracked, but not emitted anymore due
to a bug. 
>  - compaction time histogram: similar to flush time histogram, how long compactions are
taking. 
>  - compaction input num files / output num files histogram: How many files on average
we are compacting. Stripe compaction / date tiered compaction can use the num output files
metric. 
>  - compaction input / output data sizes histogram: How much data on average we are compacting.

>  - compaction input / output total bytes: Measure compaction IO / throughput. measure
write amplification, enables to set compaction throttle. 
>  - Breakdown for above for major compactions



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message