Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 16 May 2012 08:04:03 +0000 (UTC)
From: "Hadoop QA (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: 
 <172615506.3237.1337155443975.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <329277047.18550.1335989689882.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HBASE-5920) New Compactions Logic can silently
 prevent user-initiated compactions from occurring
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276567#comment-13276567 ] 

Hadoop QA commented on HBASE-5920:
----------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527581/HBASE-5920-0.92.1-1.patch
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 23 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1886//console

This message is automatically generated.
                
> New Compactions Logic can silently prevent user-initiated compactions from occurring
> ------------------------------------------------------------------------------------
>
>                 Key: HBASE-5920
>                 URL: https://issues.apache.org/jira/browse/HBASE-5920
>             Project: HBase
>          Issue Type: Bug
>          Components: client, regionserver
>    Affects Versions: 0.92.1
>            Reporter: Derek Wollenstein
>            Priority: Minor
>              Labels: compaction
>         Attachments: HBASE-5920-0.92.1-1.patch, HBASE-5920-0.92.1.patch
>
>
> There seem to be some tuning settings in which manually triggered major compactions will do nothing, including loggic
> From Store.java in the function
>   List<StoreFile> compactSelection(List<StoreFile> candidates)
> When a user manually triggers a compaction, this follows the same logic as a normal compaction check.  when a user manually triggers a major compaction, something similar happens.  Putting this all together:
> 1. If a user triggers a major compaction, this is checked against a max files threshold (hbase.hstore.compaction.max). If the number of storefiles to compact is > max files, then we downgrade to a minor compaction
> 2. If we are in a minor compaction, we do the following checks:
>    a. If the file is less than a minimum size (hbase.hstore.compaction.min.size) we automatically include it
>    b. Otherwise, we check how the size compares to the next largest size.  based on hbase.hstore.compaction.ratio.  
>   c. If the number of files included is less than a minimum count (hbase.hstore.compaction.min) then don't compact.
> In many of the exit strategies, we aren't seeing an error message.
> The net-net of this is that if we have a mix of very large and very small files, we may end up having too many files to do a major compact, but too few files to do a minor compact.
> I'm trying to go through and see if I'm understanding things correctly, but this seems like the bug
> To put it another way
> 2012-05-02 20:09:36,389 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Large Compaction requested: regionName=str,44594594594594592,1334939064521.f7aed25b55d4d7988af763bede9ce74e., store
> Name=c, fileCount=15, fileSize=1.5g (20.2k, 362.5m, 155.3k, 3.0m, 30.7k, 361.2m, 6.9m, 4.7m, 14.7k, 363.4m, 30.9m, 3.2m, 7.3k, 362.9m, 23.5m), priority=-9, time=3175046817624398; Because: Recursive enqueue; compaction_queue=(59:0), split_queue=0
> When we had a minimum compaction size of 128M, and default settings for hbase.hstore.compaction.min,hbase.hstore.compaction.max,hbase.hstore.compaction.ratio, we were not getting a compaction to run even if we ran
> major_compact 'str,44594594594594592,1334939064521.f7aed25b55d4d7988af763bede9ce74e.' from the ruby shell.  Note that we had many tiny regions (20k, 155k, 3m, 30k,..) and several large regions (362.5m,361.2m,363.4m,362.9m).  I think the bimodal nature of the sizes prevented us from doing a compaction.
> I'm not 100% sure where this errored out because when I manually triggered a compaction, I did not see
> '      // if we don't have enough files to compact, just wait             
>       if (filesToCompact.size() < this.minFilesToCompact) {              
>         if (LOG.isDebugEnabled()) {                                      
>           LOG.debug("Skipped compaction of " + this.storeNameStr         
>             + ".  Only " + (end - start) + " file(s) of size "           
>             + StringUtils.humanReadableInt(totalSize)                    
>             + " have met compaction criteria.");                         
>         }                                                                
> ' 
> being printed in the logs (and I know DEBUG logging was enabled because I saw this elsewhere).  
> I'd be happy with better error messages when we decide not to compact for user enabled compactions.
> I'd also like to see some override that says "user triggered major compaction always occurs", but maybe that's a bad idea for other reasons.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira