hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xie Gang <xiegang...@gmail.com>
Subject Re: does it make sense to get remaining space by sum all the ones of the datanode
Date Mon, 29 Jan 2018 10:19:44 GMT
2.4 and 2.6:

public long getRemaining(StorageType t) {
  long remaining = 0;
  for(DatanodeStorageInfo s : getStorageInfos()) {
    if (s.getStorageType() == t) {
      remaining += s.getRemaining();
    }
  }
  return remaining;
}


On Mon, Jan 29, 2018 at 6:12 PM, Vinayakumar B <vinayakumarb@apache.org>
wrote:

> in which version of Hadoop you are seeing this?
>
> -Vinay
>
> On 29 Jan 2018 3:26 pm, "Xie Gang" <xiegang112@gmail.com> wrote:
>
> Hello,
>
> We recently hit a issue that almost all the disk of the datanode got full
> even we configured the du .reserved.
>
> After tracking down the code, found that when we choose a target datanode
> and check if it's good candidate for block allocation (isGoodTarget()), it
> only checks if the total left space of all the volumes(the same type), not
> each volume. This logic makes the reservation of each volume useless.
> Is this a problem or do I have any misunderstanding?
>
> final long remaining = node.getRemaining(storage.getStorageType());
> if (requiredSize > remaining - scheduledSize) {
>   logNodeIsNotChosen(storage, "the node does not have enough "
>           + storage.getStorageType() + " space"
>           + " (required=" + requiredSize
>           + ", scheduled=" + scheduledSize
>           + ", remaining=" + remaining + ")");
>   stats.incrOverScheduled();
>   return false;
> }
>
>
>
> --
> Xie Gang
>



-- 
Xie Gang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message