hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Kelly (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
Date Thu, 24 Mar 2011 17:13:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010767#comment-13010767

Ivan Kelly commented on HDFS-1580:

Ah ok, since rolling has to be exposed, we should expose it fully. Your segment suggestion
sounds good. It will require a separate array in FSEditLog though, one for JournalManagers,
one for EditLogOutputStreams. I'll have to think about this. Perhaps we could get rid of the
journal manager abstraction completely and extend EditLogOutputStream to handle rolling, as
rolling is the only operation that changes for output with transactions.

Then we could have a separate class that handles stuff like format and getting a list of the
logs. I'll think about this more and put up a new design.

1. I think I mentioned the second alternative earlier in the thread, but yes, #transfer will
go away.
2. A single bookkeeper setup can be shared by many writers. The more writers you have the
slower the reads, as ledgers are interleaved when written to disk.
3. There's two layouts to think about, the storage layout and the data layout. The first will
be entirely internal to the JournalManager. The second will need to be saved though. 
4. Very good point.
5. Ok
6. Will do

> Add interface for generic Write Ahead Logging mechanisms
> --------------------------------------------------------
>                 Key: HDFS-1580
>                 URL: https://issues.apache.org/jira/browse/HDFS-1580
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ivan Kelly
>         Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf,

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message