hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1071) savenamespace should write the fsimage to all configured fs.name.dir in parallel
Date Thu, 27 May 2010 01:50:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872058#action_12872058

Konstantin Shvachko commented on HDFS-1071:

# FSImage.processIOError() and FSEditsLog.processIOError() go as a pair, sort of. If you change
prototype of one you should also change the other's. So {{FSEditsLog.processIOError()}} should
also take {{List}} rather than {{ArrayList}}.
# TestParallelImageWrite has 2 warnings (import and unused variable).
# The main question is how {{saveNamespace()}} works on a running NN. 
The scenario here is that NN grabs {{FSNamesystem}} lock, then spawns several threads, 
and starts traversing the namespace tree and write inodes to the image file. 
My understanding here is that the new threads are completely unaware of the {{FSNamesystem}}
synchronization or any other locks.
Could you please elaborate on this. I don't understand the intentions, but it doesn't look
right as is.
# It would be good to have a test case for the above condition. That is testing
saveNamespace() with multiple directories on a running mini-cluster.
# Last but not least, could you please share any performance measurements, if you have any.
How much faster save goes if the directories are 
#- on different drives, 
#- on the same drive,
#- one on disk drive and another on NFS filer.

> savenamespace should write the fsimage to all configured fs.name.dir in parallel
> --------------------------------------------------------------------------------
>                 Key: HDFS-1071
>                 URL: https://issues.apache.org/jira/browse/HDFS-1071
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HDFS-1071.2.patch, HDFS-1071.3.patch, HDFS-1071.patch
> If you have a large number of files in HDFS, the fsimage file is very big. When the namenode
restarts, it writes a copy of the fsimage to all directories configured in fs.name.dir. This
takes a long time, especially if there are many directories in fs.name.dir. Make the NN write
the fsimage to all these directories in parallel.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message