hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elaine Gan <elaine-...@gmo.jp>
Subject Re: Capacity of hdfs
Date Tue, 06 Nov 2012 08:36:06 GMT
Dear Harsh,

Thank you for the very clear explanation.
Very well noted now :)


> Hi,
> 
> Inline explanation:
> 
> On Tue, Nov 6, 2012 at 1:41 PM, Elaine Gan <elaine-gan@gmo.jp> wrote:
> > Hi,
> >
> > I'm a bit confused on the local disk and the hdfs disk.
> > Through my admin page, i see information like the following.
> >
> > Configured Capacity : 254.05 GB
> 
> Configured Capacity = Sum(Configured DN Capacity)
>                               = 64*4 = 256.
> 
> > DFS Used : 207.09 GB
> 
> HDFS files are cumulatively using 207 GB of data. This includes
> replicated blocks (default replication factor is 3).
> 
> > Non DFS Used : 16.87 GB
> 
> This is Sum(Configured DN Space) that lies outside the HDFS files.
> HDFS runs utilities such as "du"/"df" on disks and can pick up space
> that is being utilized by non HDFS, local files such as log files,
> etc. lying in the same disk as the one used for DNs.
> 
> > DFS Remaining : 30.09 GB
> 
> Your DFS has only 30 GB more to spare, which with a replication factor
> of 3, is less than 10 GB of true file length space.
> 
> > DFS Used% : 81.52 %
> > DFS Remaining% : 11.84 %
> 
> These should be self-explainatory.
> 
> > I only made configuration on the heap memory and not the disk capacity.
> > Where does "Configured Capacity : 254.05 GB " this comes from?
> > I have 4 nodes, each with 64GB of disk (local).
> > Will HDFS use it fully ?
> 
> Yes it would, unless you configure the "dfs.datanode.du.reserved"
> config param at each DN to a space value in bytes that must be left
> free on all configured volumes.
> 
> > I still need some place for local files.
> >
> > Thank you.
> >
> 
> Hope this helps!
> 
> -- 
> Harsh J



Mime
View raw message