hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2437) Refactor HLog splitLog
Date Tue, 04 May 2010 05:38:56 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12863674#action_12863674

stack commented on HBASE-2437:

Patch looks great.

+ Please change the name of this file as part of your refactoring "oldlogfile.log"
+ "logs are left in place is something goes wrong" .. should they be moved aside?
+ HBaseTestingUtility has been splintered into smaller pieces since you made this patch so
these additions of yours fit well with that general direction.
+ I love all the tests.  I like name of this thread: ZombieLastLogWriterRegionServer
+ How does this test, testLogCannotBeWrittenOnceParsed, work?  The ZombieLastLogWriterRegionServer
can only write one more edit at most (?) but seems like splitlogs proceeds from the first
through to the last as it currently does.  Why couldn't the old Zombie writer add a bunch
of edits to the last file while all other files are being split?

> Refactor HLog splitLog
> ----------------------
>                 Key: HBASE-2437
>                 URL: https://issues.apache.org/jira/browse/HBASE-2437
>             Project: Hadoop HBase
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.21.0
>            Reporter: Cosmin Lehene
>            Assignee: Cosmin Lehene
>             Fix For: 0.21.0
>         Attachments: HBASE-2437_for_HBase-0.21_with_unit_tests_for_HDFS-0.21.patch
>   Original Estimate: 120h
>  Remaining Estimate: 120h
> the HLog.splitLog got really long and complex and hard to verify for correctness. 
> I started to refactor it and also ported changes from hbase-2337 that deals with premature
deletion of log files in case of errors. Further improvements will be possible, however the
scope of this issue is to clean the code and make it behave correctly (i.e. not lose any edits)
> Added a suite of unit tests that might be ported to 0.20 as well.
> Added a setting (hbase.skip.errors - feel free to suggest a better name) that, when set
to false will make the process less tolerant to failures or corrupted files:  in case a log
file is corrupted or an error stops the process from consistently splitting the log, will
abort the entire operation to avoid losing any edits. When hbase.skip.errors is on any corrupted
files will be partially parsed and then moved to the corrupted logs archive (see hbase-2337).

> Like hbase-2337 the splitLog method will first split all the logs and then proceed to
archive them. If any splitted log file (oldlogfile.log) that is the result of an earlier splitLog
attempt is found in the region directory, it will be deleted - this is safe since we won't
move the original log files until the splitLog process completes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message