hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1108) Log newly allocated blocks
Date Wed, 24 Aug 2011 00:05:31 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089883#comment-13089883
] 

Todd Lipcon commented on HDFS-1108:
-----------------------------------

Hi Konstantin. I and several others am working on approach #2, initially -- one in which the
edit logs are accessible by both the active and standby NameNodes. Initially based on a NAS
device, but potentially also supporting other shared storage such as BookKeeper.

My understanding of the HA framework is that we are aiming to build it in such a way that
neither approach is _prevented_ - but obviously not all parts of one approach are necessary
for the other approach. In this case, this patch is useful for proposal #2 but does not prevent
approach #1. Given the APIs available in today's BackupNode, this patch is also necessary
for approach #1.

What you are describing is a potential optimization available in approach #1 that is not available
in approach #2 - and like you said, it's a nice advantage. But without this patch, _neither_
approach can be correct.

To put it another way, there are three options:
1) leave the code as it is now. That causes a potential dataloss in both approaches.
2) commit this patch. This fixes a potential dataloss in both approaches but has a small performance
impact.
3) commit a better version of this patch which also fixes dataloss but also provides a more
optimized code path for the BackupNode.

Given that the code does not exist for option 3 above, and option 1 causes dataloss regardless
of your opinions on HA, we should go with option 2 above.

Keep in mind that, even outside the scope of HA, this fixes a potential bug with NameNode
restart. If you have a small cluster that can restart quickly, this patch might allow the
NN to restart within the retry window of a client, allowing clients to more easily ride over
such restarts. Without the patch, any files in progress will be corrupted.

> Log newly allocated blocks
> --------------------------
>
>                 Key: HDFS-1108
>                 URL: https://issues.apache.org/jira/browse/HDFS-1108
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: name-node
>            Reporter: dhruba borthakur
>            Assignee: Todd Lipcon
>             Fix For: HA branch (HDFS-1623)
>
>         Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, hdfs-1108.txt
>
>
> The current HDFS design says that newly allocated blocks for a file are not persisted
in the NN transaction log when the block is allocated. Instead, a hflush() or a close() on
the file persists the blocks into the transaction log. It would be nice if we can immediately
persist newly allocated blocks (as soon as they are allocated) for specific files.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message