hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinayakumar B <vinayakum...@apache.org>
Subject Re: does it make sense to get remaining space by sum all the ones of the datanode
Date Mon, 29 Jan 2018 10:12:41 GMT
in which version of Hadoop you are seeing this?

-Vinay

On 29 Jan 2018 3:26 pm, "Xie Gang" <xiegang112@gmail.com> wrote:

Hello,

We recently hit a issue that almost all the disk of the datanode got full
even we configured the du .reserved.

After tracking down the code, found that when we choose a target datanode
and check if it's good candidate for block allocation (isGoodTarget()), it
only checks if the total left space of all the volumes(the same type), not
each volume. This logic makes the reservation of each volume useless.
Is this a problem or do I have any misunderstanding?

final long remaining = node.getRemaining(storage.getStorageType());
if (requiredSize > remaining - scheduledSize) {
  logNodeIsNotChosen(storage, "the node does not have enough "
          + storage.getStorageType() + " space"
          + " (required=" + requiredSize
          + ", scheduled=" + scheduledSize
          + ", remaining=" + remaining + ")");
  stats.incrOverScheduled();
  return false;
}



--
Xie Gang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message