hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-4054) Increase the invalidation rate based on the pending deletion count
Date Mon, 15 Oct 2012 19:26:02 GMT
Eli Collins created HDFS-4054:
---------------------------------

             Summary: Increase the invalidation rate based on the pending deletion count 

                 Key: HDFS-4054
                 URL: https://issues.apache.org/jira/browse/HDFS-4054
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Eli Collins
            Priority: Minor


It is currently possible to run out of free space on a cluster before HDFS fills up because
when deleting a file we adjust the HDFS quota immediately but don't actually delete the block
until the NN issues an invalidation to the DN (this is what pending deletions tracks). Therefore
it's possible for a cluster with a high ingest and deletion rate to fill up its local disks
even though there should be sufficient space in HDFS. One way to address this would be to
simply wait to adjust the quota until the DNs have confirmed the blocks have been deleted.
However that makes the quota tracking more complicated and doesn't help this use case. Increasing
the invalidation rate to keep up with the pending deletion count would address this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message