hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yajun Dong (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-951) DFSClient should handle all nodes in a pipeline failed.
Date Sat, 06 Feb 2010 06:28:28 GMT

    [ https://issues.apache.org/jira/browse/HDFS-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830480#action_12830480
] 

Yajun Dong commented on HDFS-951:
---------------------------------

> Also the file will leave in incomplete/being create state if that DfsClient instance
does not get a chance to close.
> But it will be not nice at all for users with respect their user experience. 
Agreed. 

I come across this problem, DFSClient should complete those creating files when IOExceptions
encoutered. 

But Normally we could not complete these files successfully before all blocks of creating
files are reported to NameNode, In short: the last block is in indeterminate state at this
time.

in this case, the only one option I think is delete the last failing block and then complete/close
the file.

> DFSClient should handle all nodes in a pipeline failed.
> -------------------------------------------------------
>
>                 Key: HDFS-951
>                 URL: https://issues.apache.org/jira/browse/HDFS-951
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: He Yongqiang
>
> processDatanodeError-> setupPipelineForAppendOrRecovery  will set streamerClosed to
be true if all nodes in the pipeline failed in the past, and just return.
> Back to run() in data streammer,  the logic 
>  if (streamerClosed || hasError || dataQueue.size() == 0 || !clientRunning) {
>                 continue;
>   }
> will just let set closed=true in closeInternal().
> And DataOutputStream will not get a chance to clean up. The DataOutputStream will throw
exception or return null for following write/close.
> It will leave the file in writing in incomplete state.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message