hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4372) Track NameNode startup progress
Date Tue, 11 Jun 2013 04:17:24 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13680190#comment-13680190
] 

Chris Nauroth commented on HDFS-4372:
-------------------------------------

Jing, thanks very much for reviewing the patch!  I'll work on a rebase.  Meanwhile, here are
responses to your feedback.

{quote}
So instead of adding extra complex counting logic to fsimage loading, do you think we can
simply set the counter of inodes to N_total after finishing loading the inodes from a fsimage?
{quote}

The desire was to have a running total of inodes loaded so far, to give a reasonable indicator
of progress up to this point.  Unfortunately, setting the counter to the total after finishing
loading would give us a very coarse progress update, which would appear to suddenly jump from
0% done to 100% done.  I wrote all of this code before snapshots, so I'll take a look to see
if there is something more reasonable to do now that snapshots are in trunk.

{quote}
Here can we present that we have not saved the namespace in the WebUI?
{quote}

I'll try to find a way to display "save not required" or similar for this case.

{quote}
Why do we need to create a step in both loadFSEdits(...) and loadEditRecords(...)?
{quote}

Since the 2 different methods can be called in isolation, I found it necessary to create a
step independently in both.  When I rebase, I'll check again to see if this is truly necessary.

{quote}
In lazyInitStep(...), maybe the code...
{quote}

I started with something similar to your suggestion, but I wanted to be conscious of instantiating
a lot of small objects that would just get discarded immediately and cause garbage collection
churn.  The problem with the alternative you suggested is that we would always call {{new
StepTracking()}} to create a new object, even if one already existed in the map.  I added
the check for {{containsKey}} to limit the impact of that.  Maybe it's just a minor optimization,
but considering that these methods could be called in tight loops during startup, I wanted
to keep it under control.

{quote}
Can we use long instead of Long for beginTime/endTime/total? We can initialize them with -1.
{quote}

I used nullable Long just because logically the values might not be initialized yet.  By switching
to primitive long with -1 reserved for "uninitialized", is the intent to reduce memory footprint
of a boxed Long?  If so, then I definitely can make that change.

                
> Track NameNode startup progress
> -------------------------------
>
>                 Key: HDFS-4372
>                 URL: https://issues.apache.org/jira/browse/HDFS-4372
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HDFS-4372.1.patch
>
>
> Track detailed progress information about the steps of NameNode startup to enable display
to users.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message