hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6423) Diskspace quota usage is wrongly updated when appending data from partial block
Date Mon, 19 May 2014 19:55:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14002258#comment-14002258

Hadoop QA commented on HDFS-6423:

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:


    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6933//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6933//console

This message is automatically generated.

> Diskspace quota usage is wrongly updated when appending data from partial block
> -------------------------------------------------------------------------------
>                 Key: HDFS-6423
>                 URL: https://issues.apache.org/jira/browse/HDFS-6423
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HDFS-6423.000.patch
> When appending new data to a file whose last block is a partial block, the diskspace
quota usage is not correctly update. For example, suppose the block size is 1024 bytes, and
a file has size 1536 bytes (1.5 blocks). If we then append another 1024 bytes to the file,
the diskspace usage for this file will not be updated to (2560 * replication) as expected,
but (2048 * replication).
> The cause of the issue is that in FSNamesystem#commitOrCompleteLastBlock, we have 
> {code}
>     // Adjust disk space consumption if required
>     final long diff = fileINode.getPreferredBlockSize() - commitBlock.getNumBytes();
>     if (diff > 0) {
>       try {
>         String path = fileINode.getFullPathName();
>         dir.updateSpaceConsumed(path, 0, -diff*fileINode.getFileReplication());
>       } catch (IOException e) {
>         LOG.warn("Unexpected exception while updating disk space.", e);
>       }
>     }
> {code}
> This code assumes that the last block of the file has never been completed before, thus
is always counted with the preferred block size in quota computation.

This message was sent by Atlassian JIRA

View raw message