hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-6401) HBase may lose edits after a crash if used with HDFS 1.0.3 or older
Date Wed, 18 Jul 2012 15:02:34 GMT

    [ https://issues.apache.org/jira/browse/HBASE-6401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417149#comment-13417149

Lars Hofhansl commented on HBASE-6401:

This needs to be fixed in a hdfs. If it is fixed in Hadoop-2 there should be a jira to backport
the change.
When I did HDFS-744 I found Hadoop-1 and Hadoop-2 quite different in the way it handled packet
shipping from the DFSClient, so the work might be non-trivial.
> HBase may lose edits after a crash if used with HDFS 1.0.3 or older
> -------------------------------------------------------------------
>                 Key: HBASE-6401
>                 URL: https://issues.apache.org/jira/browse/HBASE-6401
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.96.0
>         Environment: all
>            Reporter: nkeywal
>            Priority: Critical
>         Attachments: TestReadAppendWithDeadDN.java
> This comes from a hdfs bug, fixed in some hdfs versions. I haven't found the hdfs jira
for this.
> Context: HBase Write Ahead Log features. This is using hdfs append. If the node crashes,
the file that was written is read by other processes to replay the action.
> - So we have in hdfs one (dead) process writing with another process reading.
> - But, despite the call to syncFs, we don't always see the data when we have a dead node.
It seems to be because the call in DFSClient#updateBlockInfo ignores the ipc errors and set
the length to 0.
> - So we may miss all the writes to the last block if we try to connect to the dead DN.
> hdfs 1.0.3, branch-1 or branch-1-win: we have the issue
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/DFSClient.java?revision=1359853&view=markup
> hdfs branch-2 or trunk: we should not have the issue (but not tested)
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java?view=markup
> The attached test will fail ~50 of the time.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message