hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog
Date Fri, 29 Apr 2011 19:36:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13027159#comment-13027159

Todd Lipcon commented on HDFS-1799:

bq. JournalManager currently deals only with output
bq. Lastly, this patch doesn't hide the fact that files are in use here

These two are explicitly out of scope for this JIRA (note in the description that I call out
that this is supposed to be less ambitious than 1580). This is meant to be a start but not
an end to the refactoring of these interfaces.

bq. Regarding LogSegment, I'm still on the fence about this. I see some of the merits in it,
and I can see why JournalMananger is possibly needed for it. If you retain the list of EditLogOutputStreams
as it is now, you'll have to maintain a separate list of some other objects for the creation
of new log segments. This could be problematic as these lists need to be kept in sync for
error handling etc

Exactly what I was thinking - thanks for expressing this better than I could.

The issue is that, if an EditLogOutputStream gets an IOException, I don't want to keep using
that object. JournalManager allows us to abort that stream and get a new one next time we
start a new segment.

How would you feel about the following API as a goal:

  void setBufferCapacity(int size);
  EditLogOutputStream startLogSegment(long txId);

  void close();  // closes and "finalizes"
  void abort();  // closes but marks as possibly truncated (eg after an IOE)
  ... current calls related to write/flush/sync ...

Then, in FSEditLog, whenever we start a new segment, we'd call startLogSegment on all JournalManagers
and insert the ELOS instances into a list. Journal Managers never get marked as "failed" --
we only remove the failed ELOS after aborting it. Then on the next segment we always give
each JM a new change to start a segment.

> Refactor log rolling and filename management out of FSEditLog
> -------------------------------------------------------------
>                 Key: HDFS-1799
>                 URL: https://issues.apache.org/jira/browse/HDFS-1799
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: Edit log branch (HDFS-1073)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: Edit log branch (HDFS-1073)
>         Attachments: 0001-Added-state-management-to-FSEditLog.patch, 0002-Standardised-error-pattern.patch,
0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch, HDFS-1799-all.diff, hdfs-1799.txt,
hdfs-1799.txt, hdfs-1799.txt, hdfs-1799.txt
> This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA focuses on
pluggability, this task is simply the minimum needed for HDFS-1073:
> - Refactor the filename-specific code for rolling, diverting, and reverting log streams
out of FSEditLog into a new class
> - Clean up the related code in FSEditLog a bit
> Notably, this JIRA is going to temporarily break the BackupNode. I plan to circle back
on the BackupNode later on this branch.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message