hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1137) Name node is using the write-ahead log improperly
Date Sat, 08 May 2010 22:26:51 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865509#action_12865509

dhruba borthakur commented on HDFS-1137:

This is by design so that all transactions are not serialized on the edits log. As Todd mentions,
a client gets a response only after the transaction has been sync-ed to stable storage. if
the sync fails, and this is the only location specified in fs.name.dir, then the NameNode
actually shuts down! 

so, technically you are right that a client can see a newly file in the namespace even before
it is written to the transaction log. and if that write to the transaction log fails, then
the client would have seen a file that never will exist. This window should be small.

An alternative would be to first write to the transaction log and then update the in-memory
data structures. The performance would be impacted unless to go to a per-inode locking model.

> Name node is using the write-ahead log improperly
> -------------------------------------------------
>                 Key: HDFS-1137
>                 URL: https://issues.apache.org/jira/browse/HDFS-1137
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>            Reporter: Benjamin Reed
> The Name node is doing the write-ahead log (WAL) (aka edit log) improperly. Usually when
using WAL, changes are written to the log before they are applied to the state. Currently
the Namenode does the WAL after applying the change. This means that read may see changes
before they are durable. A client may read information and the server fail before the information
is written to the WAL, which results in the client reading state that disappears. To fix the
Namenode should write changes before (aka ahead of) applying the change.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message