hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15788) Use Offheap ByteBuffers from BufferPool to read RPC requests.
Date Tue, 08 Nov 2016 21:05:59 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648788#comment-15648788
] 

Hadoop QA commented on HBASE-15788:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 0s {color} |
{color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 4 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s {color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 39s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 9s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 36s {color}
| {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 35s {color} |
{color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s {color} | {color:green}
master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 22s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 5s {color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 35s {color}
| {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red}
The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 34m 15s {color}
| {color:green} Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5
2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 26s {color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s {color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 47s {color} | {color:green}
hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s {color} | {color:green}
hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 25s {color} | {color:red}
hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s {color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 51s {color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| Timed out junit tests | org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure
|
|   | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
|   | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
|   | org.apache.hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838024/HBASE-15788_V6.patch
|
| JIRA Issue | HBASE-15788 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  hbaseanti  checkstyle
 compile  |
| uname | Linux 3b397aae5e0f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 76814e8 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| whitespace | https://builds.apache.org/job/PreCommit-HBASE-Build/4384/artifact/patchprocess/whitespace-eol.txt
|
| unit | https://builds.apache.org/job/PreCommit-HBASE-Build/4384/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |  https://builds.apache.org/job/PreCommit-HBASE-Build/4384/artifact/patchprocess/patch-unit-hbase-server.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/4384/testReport/ |
| modules | C: hbase-common hbase-client hbase-server U: . |
| Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/4384/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Use Offheap ByteBuffers from BufferPool to read RPC requests.
> -------------------------------------------------------------
>
>                 Key: HBASE-15788
>                 URL: https://issues.apache.org/jira/browse/HBASE-15788
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>            Reporter: ramkrishna.s.vasudevan
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0
>
>         Attachments: HBASE-15788.patch, HBASE-15788_V4.patch, HBASE-15788_V5.patch, HBASE-15788_V6.patch
>
>
> Right now, when an RPC request reaches RpcServer, we read the request into an on demand
created byte[]. When it is write request and including many mutations, the request size will
be some what larger and we end up creating many temp on heap byte[] and causing more GCs.
> We have a ByteBufferPool of fixed sized off heap BBs. This is used at RpcServer while
sending read response only. We can make use of the same while reading reqs also. Instead of
reading whole of the request bytes into a single BB, we can read into N BBs (based on the
req size). When BB is not available from pool, we will fall back to old way of on demand on
heap byte[] creation.
> Remember these are off heap BBs. We read many proto objects from this read request bytes
(like header, Mutation protos etc). Thanks to PB 3 and our shading work as it supports off
heap BB now.  Also the payload cells are also in these DBBs now. The codec decoder can work
on these and create off heap BBs. Whole of our write path work with Cells now. At the time
of addition to memstore, these cells are by default copied to MSLAB ( off heap based pooled
MSLAB issue to follow this one). If MSLAB copy is not possible, we will do a copy to on heap
byte[].
> One possible down side of this is :
> Before adding to Memstore, we do write to WAL. So the Cells created out of the offheap
BBs (Codec#Decoder) will be used to write to WAL. The default FSHLog works with an OS obtained
from DFSClient. This will have only standard OS write APIs which is byte[] based.  So just
to write to WAL, we will end up in temp on heap copy for each of the Cell. The other WAL imp
(ie. AsynWAL) supports writing offheap Cells directly. We have work in progress to make AsycnWAL
as default. Also we can raise HDFS req to support BB based write APIs in their client OS?
Until then, will try for a temp workaround solution. Patch to say more on this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message