hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xie Gang <xiegang...@gmail.com>
Subject does it make sense to get remaining space by sum all the ones of the datanode
Date Mon, 29 Jan 2018 09:56:14 GMT

We recently hit a issue that almost all the disk of the datanode got full
even we configured the du .reserved.

After tracking down the code, found that when we choose a target datanode
and check if it's good candidate for block allocation (isGoodTarget()), it
only checks if the total left space of all the volumes(the same type), not
each volume. This logic makes the reservation of each volume useless.
Is this a problem or do I have any misunderstanding?

final long remaining = node.getRemaining(storage.getStorageType());
if (requiredSize > remaining - scheduledSize) {
  logNodeIsNotChosen(storage, "the node does not have enough "
          + storage.getStorageType() + " space"
          + " (required=" + requiredSize
          + ", scheduled=" + scheduledSize
          + ", remaining=" + remaining + ")");
  return false;

Xie Gang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message