hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Rodionov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14263) ExploringCompactionPolicy logic around file selection is broken
Date Thu, 27 Aug 2015 19:34:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14717369#comment-14717369
] 

Vladimir Rodionov commented on HBASE-14263:
-------------------------------------------

By setting *hbase.hstore.compaction.max.size* we limit *total* size of a minor compaction,
it makes sense when we try to minimize impact during peak hours. By setting  *hbase.hstore.compaction.min.size*
we automatically include selections lower that this value into selection candidate list, without
applying to a selection file ratio test - make sense for small new files, right after flush.

> ExploringCompactionPolicy logic around file selection is broken
> ---------------------------------------------------------------
>
>                 Key: HBASE-14263
>                 URL: https://issues.apache.org/jira/browse/HBASE-14263
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Vladimir Rodionov
>            Assignee: Vladimir Rodionov
>             Fix For: 2.0.0
>
>         Attachments: HBASE-14263.patch
>
>
> It seems that logic around selection of store file candidates is broken:
> {code}
>         // Compute the total size of files that will
>         // have to be read if this set of files is compacted.
>         long size = getTotalStoreSize(potentialMatchFiles);
>         // Store the smallest set of files.  This stored set of files will be used
>         // if it looks like the algorithm is stuck.
>         if (mightBeStuck && size < smallestSize) {
>           smallest = potentialMatchFiles;
>           smallestSize = size;
>         }
>         if (size > comConf.getMaxCompactSize()) {
>           continue;
>         }
>         ++opts;
>         if (size >= comConf.getMinCompactSize()
>             && !filesInRatio(potentialMatchFiles, currentRatio)) {
>           continue;
>         }
> {code}
> This is from applyCompactionPolicy method. As you can see, both min compaction size and
max compaction size are applied to a *selection* of files and not to individual files. It
mostly works as expected only because nobody seems using non-default hbase.hstore.compaction.max.size,
which is  Long.MAX_VALUE  and  it  is not  that  easy  to  figure out  what  is  going  on
 on an opposite side (why small files do not get included?)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message