hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3510) Improve FSEditLog pre-allocation
Date Wed, 27 Jun 2012 07:37:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402027#comment-13402027
] 

Colin Patrick McCabe commented on HDFS-3510:
--------------------------------------------

bq. TestEditLogFileOutputStream: You do not need try finally. @Before method deleted the file.

The @Before method only runs once, before all the junit tests in TestEditLogFileOutputStream
are executed.  The reason why it's there is that EditLogFileOutputStream always appends to
the end of a file; it never truncates.  So if there is garbage in the test directory before
the test begins, it's very important to remove it.  But the tests also have to clean up after
themselves, since the @Before method only runs once, not between individual junit tests. 
It makes more sense to do this in a try... finally block so that the failure of one test does
not cause all of them to fail.

I agree with the rest of the comments.  I'll add some more doxygenh and address the other
stuff
                
> Improve FSEditLog pre-allocation
> --------------------------------
>
>                 Key: HDFS-3510
>                 URL: https://issues.apache.org/jira/browse/HDFS-3510
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>             Fix For: 1.0.0, 2.0.1-alpha
>
>         Attachments: HDFS-3510-b1.001.patch, HDFS-3510-b1.002.patch, HDFS-3510.001.patch,
HDFS-3510.003.patch, HDFS-3510.004.patch, HDFS-3510.004.patch, HDFS-3510.006.patch, HDFS-3510.007.patch,
HDFS-3510.008.patch, HDFS-3510.009.patch, HDFS-3510.010.patch
>
>
> It is good to avoid running out of space in the middle of writing a batch of edits, because
when it happens, we often get partial edits at the end of the log.
> Edit log preallocation can solve this problem (see HADOOP-2330 for a full description
of edit log preallocation).
> The current pre-allocation code was introduced for performance reasons, not for preventing
partial edits.  As a consequence, we sometimes do a write without using pre-allocation.  We
should change the pre-allocation code so that it always preallocates at least enough space
before writing out the edits.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message