hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Esteban Gutierrez (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11979) Compaction progress reporting is wrong
Date Fri, 07 Nov 2014 01:03:15 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14201330#comment-14201330

Esteban Gutierrez commented on HBASE-11979:

My bad [~apurtell], got sidetracked by other things. Yeah, my guess is how we calculate the
keys for the StoreFiles:

 // NOTE: getFilterEntries could cause under-sized blooms if the user
      // switches bloom type (e.g. from ROW to ROWCOL)
      long keyCount = (r.getBloomFilterType() == store.getFamily().getBloomFilterType())
          ? r.getFilterEntries() : r.getEntries();
      fd.maxKeyCount += keyCount;


where the StoreFile can get an under estimated number of keys:

     * The number of Bloom filter entries in this store file, or an estimate
     * thereof, if the Bloom filter is not loaded. This always returns an upper
     * bound of the number of Bloom filter entries.
     * @return an estimate of the number of Bloom filter entries in this file
    public long getFilterEntries() {
      return generalBloomFilter != null ? generalBloomFilter.getKeyCount()
          : reader.getEntries();


So as we do some progress during the compaction we end above the value from BloomFilter.keycount
causing the pct to run above 100%

> Compaction progress reporting is wrong
> --------------------------------------
>                 Key: HBASE-11979
>                 URL: https://issues.apache.org/jira/browse/HBASE-11979
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Andrew Purtell
>            Assignee: Esteban Gutierrez
>            Priority: Minor
>             Fix For: 2.0.0, 0.98.8, 0.99.2
> This is a long standing problem and previously could be observed in regionserver metrics,
but, we recently added logging for long running compactions, and this has exposed the issue
in a new way, e.g.
> {noformat}
> 2014-09-15 14:20:59,450 DEBUG [regionserver8120-largeCompactions-1410813534627]
> compactions.Compactor: Compaction progress: 22683625/6808179 (333.18%), rate=162.08 kB/sec
> {noformat}
> The 'rate' reported in such logging is consistent and what we were really after, but
the progress indication is clearly broken and should be fixed.

This message was sent by Atlassian JIRA

View raw message