hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brahma Reddy Battula <bra...@apache.org>
Subject Re: about dfs.datanode.du.reserved
Date Sun, 12 Feb 2017 13:43:52 GMT
You can write a script to update this config.(Even you can manually add
this configuration in hdfs-site.xml of all the datanodes and you can
restart)



--Brahma

On Sun, Feb 12, 2017 at 12:15 PM, Alexis <alzrck@gmail.com> wrote:

> Yes you did. Thanks in advance. Is there any way to push this config to
> all the nodes from master or should I make to script to do this?
>
> Regards
>
> Enviado desde mi iPhone
>
> El 12 feb. 2017, a las 02:30, Brahma Reddy Battula <brahma@apache.org>
> escribió:
>
> Hi Alexis Fidalgo
>
> 1) I did not seen this query recently
>
> 2) you need to configure this property in slaves ( DataNode).
>
> *dfs.datanode.du.**reserved : *The number of bytes will be left free on
> the volumes used by the DataNodes. By Default,it's zero.
>
>
> For example if the disk capacity is 1TB and *dfs.datanode.du.**reserved *c
> onfigured with* 100GB.*So DataNode will not use 100GB for block
> allocation,so this data can be used by nodemanager intermittent files,
> log files .
>
> May be you can plan your MR jobs accordingly this. Hope I cleared your
> doubts.
>
> On Sat, Feb 11, 2017 at 7:26 PM, Alexis Fidalgo <alzrck@gmail.com> wrote:
>
>> Hello, i’ve tried to search archives (and google) regarding this issue
>> but had no luck. After some changes in our mapreduce code, it takes all the
>> available disk space on datanodes, before this change we hade no problem at
>> all, but since then, every few days, disks on datanodes (we have 4, all
>> with same configuration regarding disk, memory, processor, OS) becomes full
>> and we have no more mapreduce jobs completed. so i need to wipe datanodes
>> and format namenode and start all over again.
>>
>> Reading documentation i found this configuration for hdfs-site.xml
>>
>> <property>
>> <name>dfs.datanode.du.reserved</name>
>> <value>32212254720</value>
>> </property>
>> </configuration>
>>
>> Questions regarding this
>>
>> 1. is there any thread already on this issue to read and not to ask again
>> about it?
>> 2. if not 1, do i need to set up this property only on master or every
>> slave too?
>> 3. will this fix the problem or just avoid the disk become full but the
>> MR jobs will fail the same (no more space to work so we need to review our
>> code)
>>
>>
>> thanks in advance, sorry if im asking about an already discussed issue, i
>> just suscribed to the list.
>>
>> regards
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: user-help@hadoop.apache.org
>>
>>
>
>
> --
>
>
>
> --Brahma Reddy Battula
>
>


-- 



--Brahma Reddy Battula

Mime
View raw message