hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2288) Replicas awaiting recovery should return a full visible length
Date Thu, 25 Aug 2011 19:48:30 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091258#comment-13091258
] 

Todd Lipcon commented on HDFS-2288:
-----------------------------------

One simple scenario is:
- Client is writing a file with replication 0
- Client calls fsync
- the DN crashes, so the pipeline fails
- DN comes back up, and a reader tries to read the file before lease recovery has occurred

I think it's correct to return the full number of bytes on the disk, reasoning being:
- If just this DN crashed, the pipeline would have recovered before any more data was written,
and the longer replicas would have a newer generation stamp stored in the NN. If the local
DN has an older genstamp than what the client requests, it will throw an IOE in FSDataset.getReplicaVisibleLength
- If all of the DNs crashed at the same time, and the client didn't update the generation
stamp, then the replicas may have different lengths. But, all of the replicas are at least
as long as the last successful fsync, which is the only guarantee we have to provide.

Thoughts?

> Replicas awaiting recovery should return a full visible length
> --------------------------------------------------------------
>
>                 Key: HDFS-2288
>                 URL: https://issues.apache.org/jira/browse/HDFS-2288
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.23.0
>
>
> Currently, if the client calls getReplicaVisibleLength for a RWR, it returns a visible
length of 0. This causes one of HBase's tests to fail, and I believe it's incorrect behavior.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message