hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexis <alz...@gmail.com>
Subject Re: about dfs.datanode.du.reserved
Date Sun, 12 Feb 2017 06:45:28 GMT
Yes you did. Thanks in advance. Is there any way to push this config to all the nodes from
master or should I make to script to do this?

Regards 

Enviado desde mi iPhone

> El 12 feb. 2017, a las 02:30, Brahma Reddy Battula <brahma@apache.org> escribió:
> 
> Hi Alexis Fidalgo
> 
> 1) I did not seen this query recently
> 
> 2) you need to configure this property in slaves ( DataNode).
> 
> dfs.datanode.du.reserved : The number of bytes will be left free on the volumes used
by the DataNodes. By Default,it's zero.
> 
> 
> For example if the disk capacity is 1TB and dfs.datanode.du.reserved configured with
100GB.So DataNode will not use 100GB for block allocation,so this data can be used by nodemanager
intermittent files, log files .
> May be you can plan your MR jobs accordingly this. Hope I cleared your doubts.
> 
>> On Sat, Feb 11, 2017 at 7:26 PM, Alexis Fidalgo <alzrck@gmail.com> wrote:
>> Hello, i’ve tried to search archives (and google) regarding this issue but had
no luck. After some changes in our mapreduce code, it takes all the available disk space on
datanodes, before this change we hade no problem at all, but since then, every few days, disks
on datanodes (we have 4, all with same configuration regarding disk, memory, processor, OS)
becomes full and we have no more mapreduce jobs completed. so i need to wipe datanodes and
format namenode and start all over again.
>> 
>> Reading documentation i found this configuration for hdfs-site.xml
>> 
>> <property>
>> <name>dfs.datanode.du.reserved</name>
>> <value>32212254720</value>
>> </property>
>> </configuration>
>> 
>> Questions regarding this
>> 
>> 1. is there any thread already on this issue to read and not to ask again about it?
>> 2. if not 1, do i need to set up this property only on master or every slave too?
>> 3. will this fix the problem or just avoid the disk become full but the MR jobs will
fail the same (no more space to work so we need to review our code)
>> 
>> 
>> thanks in advance, sorry if im asking about an already discussed issue, i just suscribed
to the list.
>> 
>> regards
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>> 
> 
> 
> 
> -- 
> 
> 
> 
> --Brahma Reddy Battula

Mime
View raw message