hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
Date Sun, 20 Mar 2011 06:32:29 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008886#comment-13008886

Todd Lipcon commented on HDFS-1580:

Hi Ivan,

Trying to integrate this with my HDFS-1073 work, running into some things that feel wrong
about the JournalManager API...
It seems strange that every edit makes a call to getOutputStream(txid) rather than simply
calling it once at each roll point.

I think it makes the most sense to introduce the idea of a "log segment" - in the file case
it corresponds to a single edit file. In the BK case it might be a ledger. In other cases
it might be a no-op.

The state machine for logging then becomes something like:

- open
- beginLogSegment(txid 1): returns an EditLogOutputStream
- write(...) 5 times
- roll:
-- endLogSegment(1, 6)
-- startLogSegment(7)
- write more data, etc

Does this seem to make sense to you?

> Add interface for generic Write Ahead Logging mechanisms
> --------------------------------------------------------
>                 Key: HDFS-1580
>                 URL: https://issues.apache.org/jira/browse/HDFS-1580
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ivan Kelly
>         Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf,

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message