hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15180) Reduce garbage created while reading Cells from Codec Decoder
Date Tue, 16 Feb 2016 16:09:18 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148845#comment-15148845

Sean Busbey commented on HBASE-15180:

bq. UT case TestTag is in the hbase-server module which depends on hbase-common, but all changes
in the latest patch lie in hbase-common, and it seems Yetus only checked UT of the hbase-common
module and neglected the failure.

We need to make sure we run all test in all modules? Because our tests are not really UTs.
Some are (mainly those in hbase-server module) are FTs. So change in hbase-common can cause
issues with tests in hbase-server

precommit testing is meant to be a quick first-pass correctness filter (or at the least, that's
[the ideology Yetus subscribes to|http://yetus.apache.org/documentation/0.1.0/precommit-architecture/#some-philosophy]).
It's not a replacement for robust nightly checks nor committer reviews. Based on those principles,
Yetus presumes that modules are relatively self-contained wrt unit tests. In general, I think
a new feature in yetus that leveraged the same module-dependency detecting used to determine
build order in Yetus 0.2.0+ could also be used for an opt-in "build everything that depends
on module X" mode.

In our case though, precommit tests already take an absurdly long time and hbase-server is
the biggest offender (at > 1 hour per JVM). Always running the hbase-server unit tests
is simply not sustainable for us. I think a better question would be why the test coverage
in hbase-common is not sufficient to capture the behavior expected by the hbase-server module.

All that said, if you want to force a build across all modules (on the presumption that the
"change in X means also build Y" generalizes) you could touch the root pom.xml in your change,
or really any file in the top level directory.

> Reduce garbage created while reading Cells from Codec Decoder
> -------------------------------------------------------------
>                 Key: HBASE-15180
>                 URL: https://issues.apache.org/jira/browse/HBASE-15180
>             Project: HBase
>          Issue Type: Sub-task
>          Components: regionserver
>    Affects Versions: 0.98.0
>            Reporter: Anoop Sam John
>            Assignee: Anoop Sam John
>             Fix For: 2.0.0, 1.3.0, 1.2.1
>         Attachments: HBASE-15180.patch, HBASE-15180_V2.patch, HBASE-15180_V4.patch
> In KeyValueDecoder#parseCell (Default Codec decoder) we use KeyValueUtil#iscreate to
read cells from the InputStream. Here we 1st create a byte[] of length 4 and read the cell
length and then an array of Cell's length and read in cell bytes into it and create a KV.
> Actually in server we read the reqs into a byte[] and CellScanner is created on top of
a ByteArrayInputStream on top of this. By default in write path, we have MSLAB usage ON. So
while adding Cells to memstore, we will copy the Cell bytes to MSLAB memory chunks (default
2 MB size) and recreate Cells over that bytes.  So there is no issue if we create Cells over
the RPC read byte[] directly here in Decoder.  No need for 2 byte[] creation and copy for
every Cell in request.
> My plan is to make a Cell aware ByteArrayInputStream which can read Cells directly from
> Same Codec path is used in client side also. There better we can avoid this direct Cell
create and continue to do the copy to smaller byte[]s path.  Plan to introduce some thing
like a CodecContext associated with every Codec instance which can say the server/client context.

This message was sent by Atlassian JIRA

View raw message