hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
Date Thu, 06 Oct 2016 16:25:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552396#comment-15552396
] 

Rushabh S Shah commented on HDFS-10921:
---------------------------------------

bq. So the only problem that we have is the issue that Rushabh S Shah raised about Namesystem
pollution by bad tests.
I looked at the code.
{code:title=FSNamesystem.java|borderStyle=solid}
/**
   * Clear all loaded data
   */
  void clear() {
    dir.reset();
    dtSecretManager.reset();
    leaseManager.removeAllLeases();
    snapshotManager.clearSnapshottableDirs();
    cacheManager.clear();
    ecPolicyManager.clear();
    setImageLoaded(false);
    blockManager.clear();
  }
{code}
bq.     setImageLoaded(false);
This line from {{clear()}} method is the problem.

{code:title=FSDirectory.java|borderStyle=solid}
 void updateCount(INodesInPath iip, int numOfINodes,
                    QuotaCounts counts, boolean checkQuota)
                    throws QuotaExceededException {
    assert hasWriteLock();
    if (!namesystem.isImageLoaded()) {
      //still initializing. do not check or update quotas.
      return;
    }
    if (numOfINodes > iip.length()) {
      numOfINodes = iip.length();
    }
    if (checkQuota && !skipQuotaCheck) {
      verifyQuota(iip, numOfINodes, counts, null);
    }
    unprotectedUpdateCount(iip, numOfINodes, counts);
  }
{code}

{quote}
 if (!namesystem.isImageLoaded()) \{
      //still initializing. do not check or update quotas.
      return;
    \}
{quote}
While creating a directory or file, it tries to update the usageCount.
But since in the {{FSNamesystem#clear()}}, we set the imageLoaded flag to false, the {{FSDirectory#updateCount}}
will return and {{FSDirectory#unprotectedUpdateCount}} wont be called and the usage count
will not be updated.

One thing we can do is:
After calling FSNamesystem#clear in the @Before block, we can call {{cluster.getNamesystem().setImageLoaded(true);}}
I just tried running one test case {{TestDiskspaceQuotaUpdate#testQuotaIssuesWhileCommitting}}
and it passed.
Hope this helps.


> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
> --------------------------------------------------------------------
>
>                 Key: HDFS-10921
>                 URL: https://issues.apache.org/jira/browse/HDFS-10921
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Eric Badger
>            Assignee: Eric Badger
>         Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message