hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Latham (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-2770) Major compactions from shell may not major compact all families
Date Tue, 05 Jul 2011 17:19:16 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060014#comment-13060014

Dave Latham commented on HBASE-2770:

Just noticed Ted's comment on the mailing list:

Nicolas has refactored major compaction code extensively.
Can you kindly verify whether HBASE-2770 is still an issue for TRUNK ?

I don't really have time to review the current state of the compaction code and flags, but
if that code's been extensively refactored, then let's close this issue and open a new one
if it pops up again.

> Major compactions from shell may not major compact all families
> ---------------------------------------------------------------
>                 Key: HBASE-2770
>                 URL: https://issues.apache.org/jira/browse/HBASE-2770
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 0.20.4
>            Reporter: Dave Latham
>            Priority: Critical
>             Fix For: 0.92.0
> As part of a data center migration, I initiated a major_compaction request on all tables
from the shell.  A few hours later, all the region servers in the cluster appeared to have
completed the compactions and all compactionQueue metrics were back to 0.  However, some column
families of some regions had not actually done a major compaction.
> Digging through logs and code, it looks like the following happened.  The shell makes
a major compaction request which sets HRegion.forceMajorCompaction to true for every region.
 Periodically, the HRegionServer.MajorCompactionChecker checks to see if a major compaction
is needed in any family's store.  If so, calls CompactSplitThread.compactionRequested which
ends up setting the region forceMajorCompaction to false, even if it is already in the compaction
queue and set to true.  Then, when that region comes off the queue to be compacted, each family/store
separately checks for whether it should do a major compaction, so some families may not do
> (This is not good if, for example, you're doing a DistCp of the hbase dir and later on
the cluster decides to do a compaction on those files and deletes ones the DistCp job is looking
for, causing it to fail.)

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message