hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinayakumar B <vinayakum...@apache.org>
Subject Re: does it make sense to get remaining space by sum all the ones of the datanode
Date Mon, 29 Jan 2018 10:25:39 GMT
in the latest code this is is not applicable. each storage is chosen and
space of that storage is checked.

On 29 Jan 2018 3:53 pm, "Xie Gang" <xiegang112@gmail.com> wrote:

> 2.4 and 2.6:
>
> public long getRemaining(StorageType t) {
>   long remaining = 0;
>   for(DatanodeStorageInfo s : getStorageInfos()) {
>     if (s.getStorageType() == t) {
>       remaining += s.getRemaining();
>     }
>   }
>   return remaining;
> }
>
>
> On Mon, Jan 29, 2018 at 6:12 PM, Vinayakumar B <vinayakumarb@apache.org>
> wrote:
>
>> in which version of Hadoop you are seeing this?
>>
>> -Vinay
>>
>> On 29 Jan 2018 3:26 pm, "Xie Gang" <xiegang112@gmail.com> wrote:
>>
>> Hello,
>>
>> We recently hit a issue that almost all the disk of the datanode got full
>> even we configured the du .reserved.
>>
>> After tracking down the code, found that when we choose a target datanode
>> and check if it's good candidate for block allocation (isGoodTarget()), it
>> only checks if the total left space of all the volumes(the same type), not
>> each volume. This logic makes the reservation of each volume useless.
>> Is this a problem or do I have any misunderstanding?
>>
>> final long remaining = node.getRemaining(storage.getStorageType());
>> if (requiredSize > remaining - scheduledSize) {
>>   logNodeIsNotChosen(storage, "the node does not have enough "
>>           + storage.getStorageType() + " space"
>>           + " (required=" + requiredSize
>>           + ", scheduled=" + scheduledSize
>>           + ", remaining=" + remaining + ")");
>>   stats.incrOverScheduled();
>>   return false;
>> }
>>
>>
>>
>> --
>> Xie Gang
>>
>
>
>
> --
> Xie Gang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message