hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1108) Log newly allocated blocks
Date Wed, 24 Aug 2011 00:05:31 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089883#comment-13089883

Todd Lipcon commented on HDFS-1108:

Hi Konstantin. I and several others am working on approach #2, initially -- one in which the
edit logs are accessible by both the active and standby NameNodes. Initially based on a NAS
device, but potentially also supporting other shared storage such as BookKeeper.

My understanding of the HA framework is that we are aiming to build it in such a way that
neither approach is _prevented_ - but obviously not all parts of one approach are necessary
for the other approach. In this case, this patch is useful for proposal #2 but does not prevent
approach #1. Given the APIs available in today's BackupNode, this patch is also necessary
for approach #1.

What you are describing is a potential optimization available in approach #1 that is not available
in approach #2 - and like you said, it's a nice advantage. But without this patch, _neither_
approach can be correct.

To put it another way, there are three options:
1) leave the code as it is now. That causes a potential dataloss in both approaches.
2) commit this patch. This fixes a potential dataloss in both approaches but has a small performance
3) commit a better version of this patch which also fixes dataloss but also provides a more
optimized code path for the BackupNode.

Given that the code does not exist for option 3 above, and option 1 causes dataloss regardless
of your opinions on HA, we should go with option 2 above.

Keep in mind that, even outside the scope of HA, this fixes a potential bug with NameNode
restart. If you have a small cluster that can restart quickly, this patch might allow the
NN to restart within the retry window of a client, allowing clients to more easily ride over
such restarts. Without the patch, any files in progress will be corrupted.

> Log newly allocated blocks
> --------------------------
>                 Key: HDFS-1108
>                 URL: https://issues.apache.org/jira/browse/HDFS-1108
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: Todd Lipcon
>             Fix For: HA branch (HDFS-1623)
>         Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, hdfs-1108.txt
> The current HDFS design says that newly allocated blocks for a file are not persisted
in the NN transaction log when the block is allocated. Instead, a hflush() or a close() on
the file persists the blocks into the transaction log. It would be nice if we can immediately
persist newly allocated blocks (as soon as they are allocated) for specific files.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message