hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14906) Improvements on FlushLargeStoresPolicy
Date Thu, 03 Dec 2015 09:33:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037561#comment-15037561
] 

Yu Li commented on HBASE-14906:
-------------------------------

>From the HadoopQA report, observe below failures (errors) although it says +1:
{noformat}
Tests in error: 
org.apache.hadoop.hbase.regionserver.TestBulkLoad.bulkHLogShouldThrowErrorWhenFamilySpecifiedAndHFileExistsButNotInTableDescriptor(org.apache.hadoop.hbase.regionserver.TestBulkLoad)
  Run 1: TestBulkLoad.bulkHLogShouldThrowErrorWhenFamilySpecifiedAndHFileExistsButNotInTableDescriptor
� 
  Run 2: TestBulkLoad.bulkHLogShouldThrowErrorWhenFamilySpecifiedAndHFileExistsButNotInTableDescriptor
� 
  Run 3: TestBulkLoad.bulkHLogShouldThrowErrorWhenFamilySpecifiedAndHFileExistsButNotInTableDescriptor
� 

org.apache.hadoop.hbase.regionserver.TestBulkLoad.shouldThrowErrorIfBadFamilySpecifiedAsFamilyPath(org.apache.hadoop.hbase.regionserver.TestBulkLoad)
  Run 1: TestBulkLoad.shouldThrowErrorIfBadFamilySpecifiedAsFamilyPath �  Unexpected
ex...
  Run 2: TestBulkLoad.shouldThrowErrorIfBadFamilySpecifiedAsFamilyPath �  Unexpected
ex...
  Run 3: TestBulkLoad.shouldThrowErrorIfBadFamilySpecifiedAsFamilyPath �  Unexpected
ex...
{noformat}

And detailed exception:
{noformat}
shouldThrowErrorIfBadFamilySpecifiedAsFamilyPath(org.apache.hadoop.hbase.regionserver.TestBulkLoad)
 Time elapsed: 0.043 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<org.apache.hadoop.hbase.DoNotRetryIOException>
but was<java.lang.ArithmeticException>
	at org.apache.hadoop.hbase.regionserver.FlushLargeStoresPolicy.configureForRegion(FlushLargeStoresPolicy.java:59)
	at org.apache.hadoop.hbase.regionserver.FlushPolicyFactory.create(FlushPolicyFactory.java:52)
	at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:845)
	at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:786)
	at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6195)
	at org.apache.hadoop.hbase.regionserver.HRegion.createHRegion(HRegion.java:6204)
	at org.apache.hadoop.hbase.regionserver.TestBulkLoad.testRegionWithFamiliesAndSpecifiedTableName(TestBulkLoad.java:239)
	at org.apache.hadoop.hbase.regionserver.TestBulkLoad.testRegionWithFamilies(TestBulkLoad.java:249)
	at org.apache.hadoop.hbase.regionserver.TestBulkLoad.shouldThrowErrorIfBadFamilySpecifiedAsFamilyPath(TestBulkLoad.java:207)
{noformat}

Since these are Errors not Failures, the test stop at the middle phase.

The issue is caused by the patch here since it doesn't handle the case that column family
number is zero, although this won't happen in real world, it's possible in our unit test case
like TestBulkload.

> Improvements on FlushLargeStoresPolicy
> --------------------------------------
>
>                 Key: HBASE-14906
>                 URL: https://issues.apache.org/jira/browse/HBASE-14906
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 2.0.0
>            Reporter: Yu Li
>            Assignee: Yu Li
>             Fix For: 2.0.0
>
>         Attachments: HBASE-14906.patch, HBASE-14906.v2.patch
>
>
> When checking FlushLargeStoragePolicy, found below possible improving points:
> 1. Currently in selectStoresToFlush, we will do the selection no matter how many actual
families, which is not necessary for one single family
> 2. Default value for hbase.hregion.percolumnfamilyflush.size.lower.bound could not fit
in all cases, and requires user to know details of the implementation to properly set it.
We propose to use "hbase.hregion.memstore.flush.size/column_family_number" instead:
> {noformat}
>   <property>
>     <name>hbase.hregion.percolumnfamilyflush.size.lower.bound</name>
>     <value>16777216</value>
>     <description>
>     If FlushLargeStoresPolicy is used and there are multiple column families,
>     then every time that we hit the total memstore limit, we find out all the
>     column families whose memstores exceed a "lower bound" and only flush them
>     while retaining the others in memory. The "lower bound" will be
>     "hbase.hregion.memstore.flush.size / column_family_number" by default
>     unless value of this property is larger than that. If none of the families
>     have their memstore size more than lower bound, all the memstores will be
>     flushed (just as usual).
>     </description>
>   </property>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message