hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6423) Diskspace quota usage is wrongly updated when appending data to partial block
Date Wed, 21 May 2014 06:53:38 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004423#comment-14004423

Jing Zhao commented on HDFS-6423:

Thanks for the comments, Nicholas!

The issue here is that in commitOrCompleteLastBlock, when we find that the old length of the
block is >0, it is hard to determine whether the block length is derived from an fsync
call or a previous close call. The unit test testUpdateQuotaForFSync is for this scenario.
Thus in the current patch I just update the quota based on the length reported in the fsync.

> Diskspace quota usage is wrongly updated when appending data to partial block
> -----------------------------------------------------------------------------
>                 Key: HDFS-6423
>                 URL: https://issues.apache.org/jira/browse/HDFS-6423
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HDFS-6423.000.patch, HDFS-6423.001.patch
> When appending new data to a file whose last block is a partial block, the diskspace
quota usage is not correctly update. For example, suppose the block size is 1024 bytes, and
a file has size 1536 bytes (1.5 blocks). If we then append another 1024 bytes to the file,
the diskspace usage for this file will not be updated to (2560 * replication) as expected,
but (2048 * replication).
> The cause of the issue is that in FSNamesystem#commitOrCompleteLastBlock, we have 
> {code}
>     // Adjust disk space consumption if required
>     final long diff = fileINode.getPreferredBlockSize() - commitBlock.getNumBytes();
>     if (diff > 0) {
>       try {
>         String path = fileINode.getFullPathName();
>         dir.updateSpaceConsumed(path, 0, -diff*fileINode.getFileReplication());
>       } catch (IOException e) {
>         LOG.warn("Unexpected exception while updating disk space.", e);
>       }
>     }
> {code}
> This code assumes that the last block of the file has never been completed before, thus
is always counted with the preferred block size in quota computation.

This message was sent by Atlassian JIRA

View raw message