hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-5498) Improve datanode startup time
Date Mon, 18 Nov 2013 22:51:22 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13825886#comment-13825886
] 

Kihwal Lee edited comment on HDFS-5498 at 11/18/13 10:49 PM:
-------------------------------------------------------------

For quick restart (e.g. rolling upgrade), the initialization of {{BlockPoolSlice}} can be
made fater by caching {{dfsUsed}}.  Currently DU unconditionally runs the du command in its
constructor. Filed HADOOP-10111 to add an option to specify an initial "used" value and delay
the initial scan.

When I incorporated this change in {{BlockPoolSlice}}, the initialization time for each volume
dropped to several milliseconds on restart. It looks for the cache file and use the value
if it is less than 5 minutes old (the default DU interval).

Another part of start-up time improvement will come from parallelizing  the scanning &
popuating {{ReplicaMap}}.


was (Author: kihwal):
For quick restart (e.g. rolling upgrade), the initialization of {{BlockPoolSlice}} can be
made fater by cashing {{dfsUsed}}.  Currently DU unconditionally runs the du command in its
constructor. Filed HADOOP-10111 to add an option to specify an initial "used" value and delay
the initial scan.

When I incorporated this change in {{BlockPoolSlice}}, the initialization time for each volume
became several milliseconds on restart. It looks for the cache file and use the value if it
is less than 5 minutes old (the default DU interval).

Another part of start-up time improvement will come from parallelizing  the scanning &
popuating {{ReplicaMap}}.

> Improve datanode startup time
> -----------------------------
>
>                 Key: HDFS-5498
>                 URL: https://issues.apache.org/jira/browse/HDFS-5498
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kihwal Lee
>
> Similarly to HDFS-5027, an improvement  can be made for getVomeMap(). This is the phase
in which ReplicaMap.is populated.  But it will be even better if datanode scans only once
and do both.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message