hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1071) savenamespace should write the fsimage to all configured fs.name.dir in parallel
Date Thu, 24 Jun 2010 01:43:51 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12882000#action_12882000

Konstantin Shvachko commented on HDFS-1071:

So this has nothing to do with the safe mode then.
As I understand the main thread holds the global (FSNamesystem) lock, and nothing else is
going to be executed on the NN at that time. This seems to be the answer to the locking question.
Could you please add JavaDoc to {{FSImageSaver}} class with the summary of the locking and
the image identity  issues.

The approach with one serializing thread and others doing writes is in fact not harder. The
queue growth issue is not a problem imo. The speed of the total write depends on the slowest
writer, so everybody can simply wait until the slowest guy completes the assignment, they
will have to wait for him anyways in the end.
The advantage of this approach is that we guarantee everybody writes the exactly same bytes
into the image files.
With your approach, although the implementation is simpler, it is not obvious the contents
of the image files will be the same, well at least was not for me.

Out of pure curiosity if you know is there any benefit of multithreaded writing to directories
on the same drive.

> savenamespace should write the fsimage to all configured fs.name.dir in parallel
> --------------------------------------------------------------------------------
>                 Key: HDFS-1071
>                 URL: https://issues.apache.org/jira/browse/HDFS-1071
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: Dmytro Molkov
>         Attachments: HDFS-1071.2.patch, HDFS-1071.3.patch, HDFS-1071.4.patch, HDFS-1071.5.patch,
> If you have a large number of files in HDFS, the fsimage file is very big. When the namenode
restarts, it writes a copy of the fsimage to all directories configured in fs.name.dir. This
takes a long time, especially if there are many directories in fs.name.dir. Make the NN write
the fsimage to all these directories in parallel.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message