hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaoyu Yao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7658) HDFS Space Quota not working as expected
Date Fri, 30 Jan 2015 17:56:36 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298938#comment-14298938

Xiaoyu Yao commented on HDFS-7658:

HDFS doesnt know the complete size of the file ahead. It considers default blocksize ( in
your case 256MB) for calculation while adding the new block.

His default blocksize should be 128 MB without modifying dfs.blocksize in hdfs-site.xml. If
it is a 256MB block size, copy the first 10MB file will fail with RF=2 and a 500MB space quota
as you mentioned.

> HDFS Space Quota not working as expected
> ----------------------------------------
>                 Key: HDFS-7658
>                 URL: https://issues.apache.org/jira/browse/HDFS-7658
>             Project: Hadoop HDFS
>          Issue Type: Bug
>         Environment: CDH4.6
>            Reporter: Puttaswamy
> I am implementing hdfs quota in a cdh4.6 cluster .Hdfs name quota has been working properly.But
the Hdfs Space quota has not been working as expected.i.e,
> I set the space quota of 500MB for a directory say /test-space-quota.
> Then i put a file of 10 Mb into /test-space-quota which worked .Now the space available
is 480 MB ( 500 - 10*2) where 2 is rep factor.
> Then i put a file of 50Mb into /test-space-quota which worked too as expected. Now the
space available is 380 MB (480 - 50*2)
> "I am checking the quota left from the command hadoop fs -count -q /test-space-quota"
> Then i tried to put a file of 100 Mb . It should since it will just consume 200 Mb of
space with replication. But when i put that i got an error 
> "DataStreamer Exception
> org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /test
is exceeded: quota = 524288000 B = 500 MB but diskspace consumed = 662700032 B = 632 MB"
> But the quota says
> hadoop fs -count -q /test-space-quota
>         none             inf       524288000       398458880            1           
2           62914560 /test-space-quota
> Could you please help on this?

This message was sent by Atlassian JIRA

View raw message