Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 3 Aug 2011 20:11:27 +0000 (UTC)
From: "Mikhail Bautin (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: 
 <738868862.5754.1312402287786.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <696831602.25250.1304616123588.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HBASE-3857) Change the HFile Format
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078979#comment-13078979 ] 

Mikhail Bautin commented on HBASE-3857:
---------------------------------------

TL;DR version: Can we commit this patch?

@Nicolas: you mentioned some last-minute sanity checking you wanted to do.

Here are some recent test results with and without the patch. I am running the tests against the stable revision 1153300. All the sporadically failing unit tests with the patch applied (TestDistributedLogSplitting, TestHRegion, TestServerCustomProtocol) also intermittently fail without the patch. The failures look similar for the two versions of code, as shown below. The most recent version of the patch is also passing automated randomized read/write load testing. Given all of that, I think it is safe to assume that the patch does not introduce a regression. 

Note: some of these test failures might be problems with my test setup.

TestServerCustomProtocol: java.lang.AssertionError: Results should contain region test,,1312378402385.4b887abff65a2e74eab86c859d255c89. for row 'aaa'
TestDistirbutedLogSplitting:
    testThreeRSAbort(org.apache.hadoop.hbase.master.TestDistributedLogSplitting)  Time elapsed: 95.921 sec  <<< FAILURE!
    java.lang.AssertionError: 
        . . .
        at org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testThreeRSAbort
TestHRegion:
    testWritesWhileGetting(org.apache.hadoop.hbase.regionserver.TestHRegion)  Time elapsed: 0.398 sec  <<< FAILURE!
    junit.framework.AssertionFailedError: expected:<\x00\x00\x008> but was:<\x00\x00\x006>
        at org.apache.hadoop.hbase.HBaseTestCase.assertEquals(HBaseTestCase.java:691)
        at org.apache.hadoop.hbase.regionserver.TestHRegion.testWritesWhileGetting(TestHRegion.java:2759)

With the patch:

2011-08-03_02_58_34 commit: review_hfile-v2-r1153300-2011_08_02 | tests: 870, fail: 2, err: 0, skip: 9, time: 4784.9, failed: DistributedLogSplitting, HRegion
2011-08-03_03_24_45 commit: review_hfile-v2-r1153300-2011_08_02 | tests: 870, fail: 0, err: 0, skip: 9, time: 4544.1
2011-08-03_05_00_32 commit: review_hfile-v2-r1153300-2011_08_02 | tests: 870, fail: 1, err: 0, skip: 9, time: 4735.5, failed: HRegion
2011-08-03_05_21_43 commit: review_hfile-v2-r1153300-2011_08_02 | tests: 870, fail: 1, err: 0, skip: 9, time: 4485.4, failed: ServerCustomProtocol
2011-08-03_07_00_19 commit: review_hfile-v2-r1153300-2011_08_02 | tests: 870, fail: 0, err: 0, skip: 9, time: 4624.5
2011-08-03_07_14_08 commit: review_hfile-v2-r1153300-2011_08_02 | tests: 870, fail: 1, err: 0, skip: 9, time: 4687.8, failed: ServerCustomProtocol
2011-08-03_09_08_22 commit: review_hfile-v2-r1153300-2011_08_02 | tests: 870, fail: 1, err: 0, skip: 9, time: 4711.4, failed: ServerCustomProtocol

Without the patch:
2011-08-02_19_23_26 commit: HBASE-3065 don't prepend MAGIC if d | tests: 837, fail: 0, err: 0, skip: 9, time: 4496.4
2011-08-02_21_19_54 commit: HBASE-3065 don't prepend MAGIC if d | tests: 837, fail: 0, err: 0, skip: 9, time: 4520.2
2011-08-02_23_15_58 commit: HBASE-3065 don't prepend MAGIC if d | tests: 837, fail: 0, err: 0, skip: 9, time: 4438.1
2011-08-03_01_10_12 commit: HBASE-3065 don't prepend MAGIC if d | tests: 837, fail: 0, err: 0, skip: 9, time: 4503.5
2011-08-03_03_05_10 commit: HBASE-3065 don't prepend MAGIC if d | tests: 837, fail: 1, err: 0, skip: 9, time: 4550.1, failed: HRegion
2011-08-03_05_00_26 commit: HBASE-3065 don't prepend MAGIC if d | tests: 836, fail: 1, err: 0, skip: 9, time: 4418.0, failed: ServerCustomProtocol
2011-08-03_07_05_57 commit: HBASE-3065 don't prepend MAGIC if d | tests: 837, fail: 1, err: 0, skip: 9, time: 4443.9, failed: Replication
2011-08-03_08_59_20 commit: HBASE-3065 don't prepend MAGIC if d | tests: 837, fail: 2, err: 0, skip: 9, time: 4427.2, failed: CoprocessorEndpoint, DistributedLogSplitting
2011-08-03_10_52_41 commit: HBASE-3065 don't prepend MAGIC if d | tests: 837, fail: 0, err: 0, skip: 9, time: 4483.8


> Change the HFile Format
> -----------------------
>
>                 Key: HBASE-3857
>                 URL: https://issues.apache.org/jira/browse/HBASE-3857
>             Project: HBase
>          Issue Type: New Feature
>    Affects Versions: 0.90.4
>            Reporter: Liyin Tang
>            Assignee: Mikhail Bautin
>         Attachments: 0001-review_hfile-v2-r1144693_2011-07-15_11_14_44.patch, 0001-review_hfile-v2-r1147350_2011-07-26_11_55_59.patch, 0001-review_hfile-v2-r1152122-2011_08_01_03_18_00.patch, 0001-review_hfile-v2-r1153300-git-1152532-2011_08_02_19_4.patch, 0001-review_hfile-v2-r1153300-git-1152532-2011_08_03_12_4.patch, hfile_format_v2_design_draft_0.1.pdf, hfile_format_v2_design_draft_0.3.pdf, hfile_format_v2_design_draft_0.4.odt
>
>
> In order to support HBASE-3763 and HBASE-3856, we need to change the format of the HFile. The new format proposal is attached here. Thanks for Mikhail Bautin for the documentation. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira