hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1071) savenamespace should write the fsimage to all configured fs.name.dir in parallel
Date Thu, 27 May 2010 01:50:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872058#action_12872058
] 

Konstantin Shvachko commented on HDFS-1071:
-------------------------------------------

# FSImage.processIOError() and FSEditsLog.processIOError() go as a pair, sort of. If you change
prototype of one you should also change the other's. So {{FSEditsLog.processIOError()}} should
also take {{List}} rather than {{ArrayList}}.
# TestParallelImageWrite has 2 warnings (import and unused variable).
# The main question is how {{saveNamespace()}} works on a running NN. 
The scenario here is that NN grabs {{FSNamesystem}} lock, then spawns several threads, 
and starts traversing the namespace tree and write inodes to the image file. 
My understanding here is that the new threads are completely unaware of the {{FSNamesystem}}
synchronization or any other locks.
Could you please elaborate on this. I don't understand the intentions, but it doesn't look
right as is.
# It would be good to have a test case for the above condition. That is testing
saveNamespace() with multiple directories on a running mini-cluster.
# Last but not least, could you please share any performance measurements, if you have any.
How much faster save goes if the directories are 
#- on different drives, 
#- on the same drive,
#- one on disk drive and another on NFS filer.


> savenamespace should write the fsimage to all configured fs.name.dir in parallel
> --------------------------------------------------------------------------------
>
>                 Key: HDFS-1071
>                 URL: https://issues.apache.org/jira/browse/HDFS-1071
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HDFS-1071.2.patch, HDFS-1071.3.patch, HDFS-1071.patch
>
>
> If you have a large number of files in HDFS, the fsimage file is very big. When the namenode
restarts, it writes a copy of the fsimage to all directories configured in fs.name.dir. This
takes a long time, especially if there are many directories in fs.name.dir. Make the NN write
the fsimage to all these directories in parallel.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message