hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Kelly (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
Date Tue, 31 May 2011 15:32:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13041627#comment-13041627
] 

Ivan Kelly commented on HDFS-1580:
----------------------------------

Finalizing in getNumTransactions is a bit messy, because its difficult to tell if a inprogress
file is actually inprogress or the result of a crashed namenode. I tried it in HDFS-2018,
but ended up creating an explicit recoverUnclosedStreams() call, which only gets called on
loading (getNumTransactions is called in other places, such as checkpointing). It would be
possible to hide this inside the JournalManager, but then we'd have to make some sort of locking
mechanism to ensure an inprogress stream doesn't get finalised while it is writing. This is
simple for bookkeeper, but for filesystems we'd need some sort of lease mechanism.

> Add interface for generic Write Ahead Logging mechanisms
> --------------------------------------------------------
>
>                 Key: HDFS-1580
>                 URL: https://issues.apache.org/jira/browse/HDFS-1580
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ivan Kelly
>             Fix For: Edit log branch (HDFS-1073)
>
>         Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, HDFS-1580+1521.diff,
HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf,
generic_wal_iface.pdf, generic_wal_iface.txt
>
>


--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message