hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-951) DFSClient should handle all nodes in a pipeline failed.
Date Sat, 06 Feb 2010 04:37:28 GMT

    [ https://issues.apache.org/jira/browse/HDFS-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830456#action_12830456

Todd Lipcon commented on HDFS-951:

I'm not quite understanding what you're describing, I think.

If all of the datanodes fail, the last block is in an indeterminate state - we don't know
what length ever made it to the DNs, so we can't really close the file properly. I suppose
we could use the length from the last acked seqno, but the DNs will still have the replicas
in the "rbw" state. There is some kind of state transition for recovery of rbw replicas described
in the HDFS-265 document - I don't recall off the top of my head if it will function if none
of the DNs are up.

If you wait an hour for the hard lease limit, does the file end up in some kind of state that
you expect?

> DFSClient should handle all nodes in a pipeline failed.
> -------------------------------------------------------
>                 Key: HDFS-951
>                 URL: https://issues.apache.org/jira/browse/HDFS-951
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: He Yongqiang
> processDatanodeError-> setupPipelineForAppendOrRecovery  will set streamerClosed to
be true if all nodes in the pipeline failed in the past, and just return.
> Back to run() in data streammer,  the logic 
>  if (streamerClosed || hasError || dataQueue.size() == 0 || !clientRunning) {
>                 continue;
>   }
> will just let set closed=true in closeInternal().
> And DataOutputStream will not get a chance to clean up. The DataOutputStream will throw
exception or return null for following write/close.
> It will leave the file in writing in incomplete state.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message