hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Kelly (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2018) Move all journal stream management code into one place
Date Tue, 05 Jul 2011 16:29:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059988#comment-13059988
] 

Ivan Kelly commented on HDFS-2018:
----------------------------------

I've removed a lot of tests from TestFSImageStorageInspector. The functionality is now tested
by the following:


public void testCurrentStorageInspector() throws IOException {
        - Simply removed editlog stuff
   
public void testPlanWithGaps() throws IOException {
       - TestEditLog#testLoadingWithGaps() 

public void testPlanWithInProgressInMiddle() throws IOException {
       - Scenario is not possible. Inprogress files will be finalized before creating a new
segment, however this _will_ lead to the case of InconsistentEndTxIds (see below).

public void testLogGroupRecoveryNoop() throws IOException {
       - TestFileJournalManager#testNormalOperation

public void testLogGroupRecoveryMixed() throws IOException {
       - TestFileJournalManager#testInprogressRecovery
       - TestFileJournalManager#testInprogressRecoveryMixed

public void testLogGroupRecoveryInconsistentEndTxIds() throws IOException {
       - TestFileJournalManager#testManyLogsWithGapWhereInprogressInMiddle
       This isn't a failure case. If log A has 1-2 and log B has 1-5, then that means A failed
after transaction 2 and wasn't reenabled until the next roll if at all. This means that A
has a gap, from 3-5, so on loading B will be selected for that segment.

public void testLogGroupRecoveryInProgress() throws IOException {
       - TestFileJournalManager#testInprogressRecoveryAll

public void testCurrentSplitEditsAndImage() throws IOException {
       - No longer relevant whith storage inspector only handling image files

public void testLogManifest() throws IOException { 
       - TestEditLog#testEditLogManifest       

public void testLogManifestInProgressComesFirst() throws IOException { 
       - TestEditLog#testEditLogInprogressComesFirst
 

> Move all journal stream management code into one place
> ------------------------------------------------------
>
>                 Key: HDFS-2018
>                 URL: https://issues.apache.org/jira/browse/HDFS-2018
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ivan Kelly
>            Assignee: Ivan Kelly
>             Fix For: Edit log branch (HDFS-1073)
>
>         Attachments: HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff
>
>
> Currently in the HDFS-1073 branch, the code for creating output streams is in FileJournalManager
and the code for input streams is in the inspectors. This change does a number of things.
>   - Input and Output streams are now created by the JournalManager.
>   - FSImageStorageInspectors now deals with URIs when referring to edit logs
>   - Recovery of inprogress logs is performed by counting the number of transactions instead
of looking at the length of the file.
> The patch for this applies on top of the HDFS-1073 branch + HDFS-2003 patch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message