hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4372) Track NameNode startup progress
Date Tue, 11 Jun 2013 01:01:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13680112#comment-13680112
] 

Jing Zhao commented on HDFS-4372:
---------------------------------

Hi Chris,

   The patch looks very good to me. Currently the patch needs rebase (mainly for the changes
in FSImageFormat.java). Some thoughts about the patch:

1. FSImageFormat.java
After snapshot is supported, the number of files/directories that is read in the head of the
fsimage (denoted as N_total), may no longer be equal to the number of times of invocation
of loadINode(...). This is because when we compute the total namespace quota usage, some nodes
may be counted multiple times (in case that rename happened with existence of snapshots).
Thus when using a counter to track the number of inodes loaded from the fsimage, it is challenging
to keep the final value of the counter equal to N_total.

So instead of adding extra complex counting logic to fsimage loading, do you think we can
simply set the counter of inodes to N_total after finishing loading the inodes from a fsimage?

2. FSNamesystem.java, in loadFSImage(...),

{code}
+      } else {
+        // No need to save, so mark the phase done.
+        StartupProgress prog = NameNode.getStartupProgress();
+        prog.beginPhase(Phase.SAVING_CHECKPOINT);
+        prog.endPhase(Phase.SAVING_CHECKPOINT);
{code}

Here can we present that we have not saved the namespace in the WebUI? (Users may also get
the same information when they see a very small elapsed time for the phase).

Some minor issues:

3. FSEditLogLoader.java
Why do we need to create a step in both loadFSEdits(...) and loadEditRecords(...)?

4. StartupProgress.java
4.1 There are some unused imports.

4.2 In lazyInitStep(...), maybe the code 
{code}
  if (!steps.containsKey(step)) {
    steps.putIfAbsent(step, new StepTracking());
  }
  return steps.get(step);
{code}

can be simplified to 

{code}
  steps.putIfAbsent(step, new StepTracking());
  return steps.get(step);
{code}

or something like 

{code}
  StepTracking st = new StepTracking();
  StepTracking oldValue =  steps.putIfAbsent(step, st);
  return oldValue != null ? oldValue : st;
{code}

5. StepTracking.java and PhaseTracking.java
Can we use long instead of Long for beginTime/endTime/total? We can initialize them with -1.
                
> Track NameNode startup progress
> -------------------------------
>
>                 Key: HDFS-4372
>                 URL: https://issues.apache.org/jira/browse/HDFS-4372
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-4372.1.patch
>
>
> Track detailed progress information about the steps of NameNode startup to enable display
to users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message