hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manoj Samel <manojsamelt...@gmail.com>
Subject Re: DFS Used V/S Non DFS Used
Date Fri, 10 Oct 2014 21:38:29 GMT
Thanks Suresh - still not clear

Say the "dfs.datanode.du.reserved" is not set (default seems 0). The "non
DFS Used" reported number is non-zero. What does this means ? What is being
referred as "temp files" ? and how can they encroach in the example of
/disk1/datanode, /disk2/datanode etc.

Thanks,

On Fri, Oct 10, 2014 at 2:29 PM, Suresh Srinivas <suresh@hortonworks.com>
wrote:

> Here is the information from -
> https://issues.apache.org/jira/browse/HADOOP-4430?focusedCommentId=12640259&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12640259
> Here are the definition of data reported on the Web UI:
> Configured Capacity: Disk space corresponding to all the data directories
> - Reserved space as defined by dfs.datanode.du.reserved
> DFS Used: Space used by DFS
> Non DFS Used: 0 if the temporary files do not exceed reserved space.
> Otherwise this is the size by which temporary files exceed the reserved
> space and encroach into the DFS configured space.
> DFS Remaining: (Configured Capacity - DFS Used - Non DFS Used)
> DFS Used %: (DFS Used / Configured Capacity) * 100
> DFS Remaining % = (DFS Remaining / Configured Capacity) * 100
>
> On Fri, Oct 10, 2014 at 2:21 PM, Manoj Samel <manojsameltech@gmail.com>
> wrote:
>
>> Hi,
>>
>> Not clear how this computation is done
>>
>> For sake of discussion Say the machine with data node has two disks
>> /disk1 and /disk2. And each of these disk has a directory for data node and
>> a directory for non-datanode usage.
>>
>> /disk1/datanode
>> /disk1/non-datanode
>> /disk2/datanode
>> /disk2/non-datanode
>>
>> The dfs.datanode.data.dir says "/disk1/datanode,/disk2/datanode".
>>
>> With this, what does the DFS and NonDFS indicates? Does it indicates
>> SUM(/disk*/datanode) & SUM(/disk*/non-datanode) etc. resp. ?
>>
>> Thanks,
>>
>>
>>
>
>
> --
> http://hortonworks.com/download/
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message