hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7943) Append cannot handle the last block with length greater than the preferred block size
Date Wed, 18 Mar 2015 21:25:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367953#comment-14367953

Jing Zhao commented on HDFS-7943:

I kindly of have an initial patch fixing the issue following #1. Then I find that more places
have the assumption that the block's size upper limit is the preferred size. E.g., when conversion
between UC block and complete block happens, we always follow this assumption to update quota
usage. Also, when computing storage usage of a UC file, we use the preferred size as the size
of the last UC block. These places require fixes as well.

Currently I prefer #2 because of its simplicity. Also I think it is good to have the preferred
block size as the upper limit. This upper limit is also kind of respected by the client while
writing data.

What do you think, [~szetszwo]?

> Append cannot handle the last block with length greater than the preferred block size
> -------------------------------------------------------------------------------------
>                 Key: HDFS-7943
>                 URL: https://issues.apache.org/jira/browse/HDFS-7943
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.7.0
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>            Priority: Blocker
>         Attachments: HDFS-7943.000.patch
> In HDFS-3689, we remove the restriction from concat that all the source files should
have the same preferred block size with the target file. This can cause a file to contain
blocks with size larger than its preferred block size.
> If such block happens to be the last block of a file, and later we append data to the
file without the {{CreateFlag.NEW_BLOCK}} flag (i.e., appending data to the last block), looks
like the current client code will keep writing to the last block and never allocate a new

This message was sent by Atlassian JIRA

View raw message