hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
Date Tue, 25 Jan 2011 21:57:45 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986709#action_12986709
] 

Jitendra Nath Pandey commented on HDFS-1580:
--------------------------------------------

   The interface also needs to have a counterpart of roll-edits method. Currently, for checkpointing,
first thing being done is to roll the edit logs i.e. an edits.new is created. As I understand,
in hdfs-1073 instead of edits.new the edit files will be numbered. At least FileWriteAheadLog
will need to roll to keep edit files from getting too big, even if it is not required for
checkpointing. 
   
   The interface should also provide methods to get all previously rotated edit log files
(or ledgers) and also current "in-progress" edit log file or ledger. 

   As a suggestion, the interface could have a concept of log handles, where each handle uniquely
corresponds to single edit log file or ledger. Thus, we could have a method getAllLogs and
it will return a list of log-handles.
I think ordered handles will fit with hdfs-1073 model (need to confirm). LogHandle can also
have some meta data for example first transaction id, whether its current or old etc. Hdfs-1073
is proposing to store first transaction-id in the edit-file name itself, which could be used
to populate the log-handle in case of FileWriteAheadLog. The input and output streams should
be in the LogHandle, so that any log-file can be read. Log-Handle for older files should not
let one create an output stream.  

  A method to purge the editlogs might also be needed, i.e. given a handle remove the corresponding
log-file (or ledger).



> Add interface for generic Write Ahead Logging mechanisms
> --------------------------------------------------------
>
>                 Key: HDFS-1580
>                 URL: https://issues.apache.org/jira/browse/HDFS-1580
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Ivan Kelly
>         Attachments: generic_wal_iface.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message