hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1780) reduce need to rewrite fsimage on statrtup
Date Thu, 07 Jul 2011 13:48:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061307#comment-13061307
] 

Daryn Sharp commented on HDFS-1780:
-----------------------------------

Very nice, a few points/suggestions:
* Multiply {{checkpointPeriod}} by 1000 during assignment to make the following comparison
use consistent units.  Ex. The first thought in my mind was to make sure {{checkpointAge}}
didn't need to be multiplied too.
* Did you perhaps miss including {{DFSConfigKeys.java}} in this patch?  I neither see {{DFSConfigKeys.DFS_NAMENODE_CHECKPOINT_TXNS_KEY}}
in the source tree, nor added in this patch.  
* Should the tests have an assert to verify whether it should or shouldn't have re-checkpointed?
 I would expect a second testcase to be added to verify it does or does not checkpoint, or
is that implicitly tested by the existing cases?

> reduce need to rewrite fsimage on statrtup
> ------------------------------------------
>
>                 Key: HDFS-1780
>                 URL: https://issues.apache.org/jira/browse/HDFS-1780
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: Edit log branch (HDFS-1073)
>            Reporter: Daryn Sharp
>            Assignee: Todd Lipcon
>             Fix For: Edit log branch (HDFS-1073)
>
>         Attachments: hdfs-1780.txt, hdfs-1780.txt
>
>
> On startup, the namenode will read the fs image, apply edits, then rewrite the fs image.
 This requires a non-trivial amount of time for very large directory structures.  Perhaps
the namenode should employ some logic to decide that the edits are simple enough that it doesn't
warrant rewriting the image back out to disk.
> A few ideas:
> Use the size of the edit logs, if the size is below a threshold, assume it's cheaper
to reprocess the edit log instead of writing the image back out.
> Time the processing of the edits and if the time is below a defined threshold, the image
isn't rewritten.
> Timing the reading of the image, and the processing of the edits.  Base the decision
on the time it would take to write the image (a multiplier is applied to the read time?) versus
the time it would take to reprocess the edits.  If a certain threshold (perhaps percentage
or expected time to rewrite) is exceeded, rewrite the image.
> Somethingalong the lines of the last suggestion may allow for defaults that adapt for
any size cluster, thus eliminating the need to keep tweaking a cluster's settings based on
its size.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message