hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2018) 1073: Move all journal stream management code into one place
Date Thu, 25 Aug 2011 23:26:30 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091414#comment-13091414
] 

Jitendra Nath Pandey commented on HDFS-2018:
--------------------------------------------

> 3. Todd raised some concerns about removing the caching introduced in HDFS-2018, Sanjay
and Jitendra are going to 
> take a look and comment.
   I discussed with Sanjay and Suresh regarding the caching of edit log file names in FileJournalManager.
We agreed that the cache is not really needed in the FileJournalManager, because there is
not any significant performance concern. The cache for list of valid editlog files in FileJournalManager
is used only for recovery for a read. I think this cache can be removed once explicit recoverLogs
API is introduced. This should address Todd's concern.

  So the following should let us reach an agreeable patch:
  1. Modify Ivan's patch to introduce recoverLogs API.
  2. The point (1) above will let us remove the cache from FileJournalManager.  
  3. As agreed before getEditLogManifest should be moved to JournalAdminInterface, but we
can leave it for later as mentioned in Eli's comment, so no changes for this in HDFS-2018.


> 1073: Move all journal stream management code into one place
> ------------------------------------------------------------
>
>                 Key: HDFS-2018
>                 URL: https://issues.apache.org/jira/browse/HDFS-2018
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Ivan Kelly
>            Assignee: Ivan Kelly
>             Fix For: 0.23.0
>
>         Attachments: HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
hdfs-2018-otherapi.txt, hdfs-2018.txt
>
>
> Currently in the HDFS-1073 branch, the code for creating output streams is in FileJournalManager
and the code for input streams is in the inspectors. This change does a number of things.
>   - Input and Output streams are now created by the JournalManager.
>   - FSImageStorageInspectors now deals with URIs when referring to edit logs
>   - Recovery of inprogress logs is performed by counting the number of transactions instead
of looking at the length of the file.
> The patch for this applies on top of the HDFS-1073 branch + HDFS-2003 patch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message