hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhanwei Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7885) Datanode should not trust the generation stamp provided by client
Date Thu, 05 Mar 2015 02:05:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347963#comment-14347963

Zhanwei Wang commented on HDFS-7885:

In function {{getBlockLocalPathInfo}} the input parameter {{block}} is passed by the client.
Since client will buffer file's metadata, block.getGenerationStamp() may be older then the
real generationStamp on Datanode. Datanode will report that it cannot find metadata file and
then client fail to read.

@Override // FsDatasetSpi
  public BlockLocalPathInfo getBlockLocalPathInfo(ExtendedBlock block)
      throws IOException {
    File datafile = getBlockFile(block);
    File metafile = FsDatasetUtil.getMetaFile(datafile, block.getGenerationStamp());
    BlockLocalPathInfo info = new BlockLocalPathInfo(block,
        datafile.getAbsolutePath(), metafile.getAbsolutePath());
    return info;

Test case

enable read-circuit and set {{dfs.client.use.legacy.blockreader.local}} to true
1) crete a file with two blocks.
2) open it for read, but not read. (client fetch block metadata)
3) append to it. (increase generation stamp of last block)
4) continue to read. (will fail)

> Datanode should not trust the generation stamp provided by client
> -----------------------------------------------------------------
>                 Key: HDFS-7885
>                 URL: https://issues.apache.org/jira/browse/HDFS-7885
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.2.0
>            Reporter: vitthal (Suhas) Gogate
>            Priority: Critical
> Datanode should not trust the generation stamp provided by client, since it is prefetched
and buffered in client, and concurrent append may increase it.

This message was sent by Atlassian JIRA

View raw message