hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "churro morales (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-11322) SnapshotHFileCleaner makes the wrong check for lastModified time thus causing too many cache refreshes
Date Tue, 10 Jun 2014 22:37:02 GMT
churro morales created HBASE-11322:
--------------------------------------

             Summary: SnapshotHFileCleaner makes the wrong check for lastModified time thus
causing too many cache refreshes
                 Key: HBASE-11322
                 URL: https://issues.apache.org/jira/browse/HBASE-11322
             Project: HBase
          Issue Type: Bug
    Affects Versions: 0.94.19
            Reporter: churro morales
            Assignee: churro morales
            Priority: Critical


In the SnapshotFileCache:
The last modified time is done via this operation:
{code}
this.lastModifiedTime = Math.min(dirStatus.getModificationTime(),
                                     tempStatus.getModificationTime());
{code}

and the check to see if the snapshot directories have been modified:
{code}
// if the snapshot directory wasn't modified since we last check, we are done
    if (dirStatus.getModificationTime() <= lastModifiedTime &&
        tempStatus.getModificationTime() <= lastModifiedTime) {
      return;
    }
{code}

so if the dirStatus and tmpStatus are modified at different times, we will always assume they
have been modified and refresh the cache.

In our cluster, this was a huge performance hit.  The cleaner chain fell behind, thus almost
filling up dfs and our namenode heap.

Its a simple fix, instead of Math.min we use Math.max for the lastModified, I believe that
will be correct.

I'll apply a patch for you guys.





--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message