hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-8665) bad compaction priority behavior in queue can cause store to be blocked
Date Thu, 06 Jun 2013 22:20:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-8665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13677595#comment-13677595
] 

Hadoop QA commented on HBASE-8665:
----------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12586586/HBASE-8665-v0.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 5 new or modified
tests.

    {color:green}+1 hadoop1.0{color}.  The patch compiles against the hadoop 1.0 profile.

    {color:green}+1 hadoop2.0{color}.  The patch compiles against the hadoop 2.0 profile.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:red}-1 release audit{color}.  The applied patch generated 1 release audit warnings
(more than the trunk's current 0 warnings).

    {color:green}+1 lineLengths{color}.  The patch does not introduce lines longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

     {color:red}-1 core tests{color}.  The patch failed these unit tests:
                       org.apache.hadoop.hbase.TestFullLogReconstruction

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5963//console

This message is automatically generated.
                
> bad compaction priority behavior in queue can cause store to be blocked
> -----------------------------------------------------------------------
>
>                 Key: HBASE-8665
>                 URL: https://issues.apache.org/jira/browse/HBASE-8665
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>         Attachments: HBASE-8665-v0.patch
>
>
> Note that this can be solved by bumping up the number of compaction threads but still
it seems like this priority "inversion" should be dealt with.
> There's a store with 1 big file and 3 flushes (1 2 3 4) sitting around and minding its
own business when it decides to compact. Compaction (2 3 4) is created and put in queue, it's
low priority, so it doesn't get out of the queue for some time - other stores are compacting.
Meanwhile more files are flushed and at (1 2 3 4 5 6 7) it decides to compact (5 6 7). This
compaction now has higher priority than the first one. After that if the load is high it enters
vicious cycle of compacting and compacting files as they arrive, with store being blocked
on and off, with the (2 3 4) compaction staying in queue for up to ~20 minutes (that I've
seen).
> I wonder why we do thing thing where we queue compaction and compact separately. Perhaps
we should take snapshot of all store priorities, then do select in order and execute the first
compaction we find. This will need starvation safeguard too but should probably be better.
> Btw, exploring compaction policy may be more prone to this, as it can select files from
the middle, not just beginning, which, given the treatment of already selected files that
was not changed from the old ratio-based one (all files with lower seqNums than the ones selected
are also ineligible for further selection), will make more files ineligible (e.g. imagine
with 10 blocking files, with 8 present (1-8), (6 7 8) being selected and getting stuck). Today
I see the case that would also apply to old policy, but yesterday I saw file distribution
something like this: 4,5g, 2,1g, 295,9m, 113,3m, 68,0m, 67,8m, 1,1g, 295,1m, 100,4m, unfortunately
w/o enough logs to figure out how it resulted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message