Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Message-ID: <29405996.547371282696401288.JavaMail.jira@thor>
Date: Tue, 24 Aug 2010 20:33:21 -0400 (EDT)
From: "Nicolas Spiegelberg (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Subject: [jira] Commented: (HBASE-2643) Figure how to deal with eof
 splitting logs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902206#action_12902206 ] 

Nicolas Spiegelberg commented on HBASE-2643:
--------------------------------------------

IIIRC:

[5:09pm] nspiegelberg: I'm just wondering if there's a reason for us to not continue through an EOF
[5:09pm] St^Ack: I can't think of one
[5:10pm] St^Ack: what you say is reasonable nspiegelberg
[5:10pm] St^Ack: least till we learn otherwise
[5:10pm] St^Ack: I have to go lads... pick up kid
[5:10pm] kannan: i agree... we could check to see if the IOE was an EOF and continue on....

> Figure how to deal with eof splitting logs
> ------------------------------------------
>
>                 Key: HBASE-2643
>                 URL: https://issues.apache.org/jira/browse/HBASE-2643
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Nicolas Spiegelberg
>            Priority: Blocker
>             Fix For: 0.90.0
>
>
> When splitting the WAL and encountering EOF, it's not clear what to do. Initial discussion of this started in http://review.hbase.org/r/74/ - summarizing here for brevity:
> We can get an EOFException while splitting the WAL in the following cases:
> - The writer died after creating the file but before even writing the header (or crashed halfway through writing the header)
> - The writer died in the middle of flushing some data - sync() guarantees that we can see _at least_ the last edit, but we may see half of an edit that was being written out when the RS crashed (especially for large rows)
> - The data was actually corrupted somehow (eg a length field got changed to be too long and thus points past EOF)
> Ideally we would know when we see EOF whether it was really the last record, and in that case, simply drop that record (it wasn't synced, so therefore we dont need to split it). Some open questions:
>   - Currently we ignore empty files. Is it ok to ignore an empty log file if it's not the last one?
>   - Similarly, do we ignore an EOF mid-record if it's not the last log file?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.