Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 6 Oct 2016 16:25:20 +0000 (UTC)
From: "Rushabh S Shah (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13008246.1475072668000.754471.1475771120561@Atlassian.JIRA>
In-Reply-To: <JIRA.13008246.1475072668000@Atlassian.JIRA>
References: <JIRA.13008246.1475072668000@Atlassian.JIRA> <JIRA.13008246.1475072668652@arcas>
Subject: [jira] [Commented] (HDFS-10921) TestDiskspaceQuotaUpdate doesn't
 wait for NN to get out of safe mode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 06 Oct 2016 16:25:22 -0000


    [ https://issues.apache.org/jira/browse/HDFS-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15552396#comment-15552396 ] 

Rushabh S Shah commented on HDFS-10921:
---------------------------------------

bq. So the only problem that we have is the issue that Rushabh S Shah raised about Namesystem pollution by bad tests.
I looked at the code.
{code:title=FSNamesystem.java|borderStyle=solid}
/**
   * Clear all loaded data
   */
  void clear() {
    dir.reset();
    dtSecretManager.reset();
    leaseManager.removeAllLeases();
    snapshotManager.clearSnapshottableDirs();
    cacheManager.clear();
    ecPolicyManager.clear();
    setImageLoaded(false);
    blockManager.clear();
  }
{code}
bq.     setImageLoaded(false);
This line from {{clear()}} method is the problem.

{code:title=FSDirectory.java|borderStyle=solid}
 void updateCount(INodesInPath iip, int numOfINodes,
                    QuotaCounts counts, boolean checkQuota)
                    throws QuotaExceededException {
    assert hasWriteLock();
    if (!namesystem.isImageLoaded()) {
      //still initializing. do not check or update quotas.
      return;
    }
    if (numOfINodes > iip.length()) {
      numOfINodes = iip.length();
    }
    if (checkQuota && !skipQuotaCheck) {
      verifyQuota(iip, numOfINodes, counts, null);
    }
    unprotectedUpdateCount(iip, numOfINodes, counts);
  }
{code}

{quote}
 if (!namesystem.isImageLoaded()) \{
      //still initializing. do not check or update quotas.
      return;
    \}
{quote}
While creating a directory or file, it tries to update the usageCount.
But since in the {{FSNamesystem#clear()}}, we set the imageLoaded flag to false, the {{FSDirectory#updateCount}} will return and {{FSDirectory#unprotectedUpdateCount}} wont be called and the usage count will not be updated.

One thing we can do is:
After calling FSNamesystem#clear in the @Before block, we can call {{cluster.getNamesystem().setImageLoaded(true);}}
I just tried running one test case {{TestDiskspaceQuotaUpdate#testQuotaIssuesWhileCommitting}} and it passed.
Hope this helps.


> TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode
> --------------------------------------------------------------------
>
>                 Key: HDFS-10921
>                 URL: https://issues.apache.org/jira/browse/HDFS-10921
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Eric Badger
>            Assignee: Eric Badger
>         Attachments: HDFS-10921.001.patch, HDFS-10921.002.patch
>
>
> Test fails intermittently because the NN is still in safe mode. 


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org