Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1E7F317E98 for ; Thu, 28 Jan 2016 18:19:20 +0000 (UTC) Received: (qmail 69358 invoked by uid 500); 28 Jan 2016 18:12:40 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 69311 invoked by uid 500); 28 Jan 2016 18:12:40 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 69296 invoked by uid 99); 28 Jan 2016 18:12:40 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 28 Jan 2016 18:12:40 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 0D8A92C1F57 for ; Thu, 28 Jan 2016 18:12:40 +0000 (UTC) Date: Thu, 28 Jan 2016 18:12:40 +0000 (UTC) From: "Hadoop QA (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HBASE-13259) mmap() based BucketCache IOEngine MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122009#comment-15122009 ] Hadoop QA commented on HBASE-13259: ----------------------------------- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} | {color:green} Patch does not have any anti-patterns. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 16s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s {color} | {color:green} master passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s {color} | {color:green} master passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m 20s {color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 32s {color} | {color:green} master passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 59s {color} | {color:red} hbase-common in master has 1 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 23s {color} | {color:red} hbase-server in master has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s {color} | {color:green} master passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s {color} | {color:green} master passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s {color} | {color:green} the patch passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 12s {color} | {color:red} Patch generated 2 new checkstyle issues in hbase-common (total was 7, now 9). {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 25s {color} | {color:red} Patch generated 4 new checkstyle issues in hbase-server (total was 36, now 40). {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 33s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} | | {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 30m 57s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s {color} | {color:green} the patch passed with JDK v1.8.0_72 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s {color} | {color:green} hbase-common in the patch passed with JDK v1.8.0_72. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 149m 35s {color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 27s {color} | {color:green} hbase-common in the patch passed with JDK v1.7.0_91. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 109m 29s {color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 1s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 333m 2s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_72 Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer | | | hadoop.hbase.mapreduce.TestTableInputFormatScan2 | | | hadoop.hbase.replication.TestReplicationSmallTests | | JDK v1.8.0_72 Timed out junit tests | org.apache.hadoop.hbase.client.TestAdmin2 | | | org.apache.hadoop.hbase.client.TestAdmin1 | | | org.apache.hadoop.hbase.client.TestHCM | | | org.apache.hadoop.hbase.client.TestSnapshotFromClientWithRegionReplicas | | JDK v1.7.0_91 Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer | | JDK v1.7.0_91 Timed out junit tests | org.apache.hadoop.hbase.util.TestHBaseFsckOneRS | | | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd | \\ \\ || Subsystem || Report/Notes || | Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-01-28 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12784909/HBASE-13259_v3.patch | | JIRA Issue | HBASE-13259 | | Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile | | uname | Linux 0f485e2d9ccc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh | | git revision | master / 47c4147 | | findbugs | v3.0.0 | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/337/artifact/patchprocess/branch-findbugs-hbase-common-warnings.html | | findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/337/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html | | checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/337/artifact/patchprocess/diff-checkstyle-hbase-common.txt | | checkstyle | https://builds.apache.org/job/PreCommit-HBASE-Build/337/artifact/patchprocess/diff-checkstyle-hbase-server.txt | | whitespace | https://builds.apache.org/job/PreCommit-HBASE-Build/337/artifact/patchprocess/whitespace-eol.txt | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/337/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt | | unit | https://builds.apache.org/job/PreCommit-HBASE-Build/337/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_91.txt | | unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/337/artifact/patchprocess/patch-unit-hbase-server-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HBASE-Build/337/artifact/patchprocess/patch-unit-hbase-server-jdk1.7.0_91.txt | | JDK v1.7.0_91 Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/337/testReport/ | | modules | C: hbase-common hbase-server U: . | | Max memory used | 338MB | | Powered by | Apache Yetus 0.1.0 http://yetus.apache.org | | Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/337/console | This message was automatically generated. > mmap() based BucketCache IOEngine > --------------------------------- > > Key: HBASE-13259 > URL: https://issues.apache.org/jira/browse/HBASE-13259 > Project: HBase > Issue Type: New Feature > Components: BlockCache > Affects Versions: 0.98.10 > Reporter: Zee Chen > Assignee: Zee Chen > Priority: Critical > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, HBASE-13259_v3.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, mmap-trunk-v1.patch > > > Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data from kernel space to user space. This is a good choice when the total working set size is much bigger than the available RAM and the latency is dominated by IO access. However, when the entire working set is small enough to fit in the RAM, using mmap() (and subsequent memcpy()) to move data from kernel space to user space is faster. I have run some short keyval gets tests and the results indicate a reduction of 2%-7% of kernel CPU on my system, depending on the load. On the gets, the latency histograms from mmap() are identical to those from pread(), but peak throughput is close to 40% higher. > This patch modifies ByteByfferArray to allow it to specify a backing file. > Example for using this feature: set hbase.bucketcache.ioengine to mmap:/dev/shm/bucketcache.0 in hbase-site.xml. > Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)