hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4372) Track NameNode startup progress
Date Tue, 11 Jun 2013 04:17:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13680190#comment-13680190

Chris Nauroth commented on HDFS-4372:

Jing, thanks very much for reviewing the patch!  I'll work on a rebase.  Meanwhile, here are
responses to your feedback.

So instead of adding extra complex counting logic to fsimage loading, do you think we can
simply set the counter of inodes to N_total after finishing loading the inodes from a fsimage?

The desire was to have a running total of inodes loaded so far, to give a reasonable indicator
of progress up to this point.  Unfortunately, setting the counter to the total after finishing
loading would give us a very coarse progress update, which would appear to suddenly jump from
0% done to 100% done.  I wrote all of this code before snapshots, so I'll take a look to see
if there is something more reasonable to do now that snapshots are in trunk.

Here can we present that we have not saved the namespace in the WebUI?

I'll try to find a way to display "save not required" or similar for this case.

Why do we need to create a step in both loadFSEdits(...) and loadEditRecords(...)?

Since the 2 different methods can be called in isolation, I found it necessary to create a
step independently in both.  When I rebase, I'll check again to see if this is truly necessary.

In lazyInitStep(...), maybe the code...

I started with something similar to your suggestion, but I wanted to be conscious of instantiating
a lot of small objects that would just get discarded immediately and cause garbage collection
churn.  The problem with the alternative you suggested is that we would always call {{new
StepTracking()}} to create a new object, even if one already existed in the map.  I added
the check for {{containsKey}} to limit the impact of that.  Maybe it's just a minor optimization,
but considering that these methods could be called in tight loops during startup, I wanted
to keep it under control.

Can we use long instead of Long for beginTime/endTime/total? We can initialize them with -1.

I used nullable Long just because logically the values might not be initialized yet.  By switching
to primitive long with -1 reserved for "uninitialized", is the intent to reduce memory footprint
of a boxed Long?  If so, then I definitely can make that change.

> Track NameNode startup progress
> -------------------------------
>                 Key: HDFS-4372
>                 URL: https://issues.apache.org/jira/browse/HDFS-4372
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-4372.1.patch
> Track detailed progress information about the steps of NameNode startup to enable display
to users.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message