hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8818) Allow Balancer to run faster
Date Tue, 02 May 2017 20:19:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993654#comment-15993654

Kihwal Lee commented on HDFS-8818:

-1. Proposing revert of this untested change.

We've run it on a small (to us) 280 node cluster.  It moves a little for a couple of iterations
and hangs forever.  So, contrary to the title, it becomes extremely slow. Blindly creating
a thread pool per datanode is a bad idea.

  public void executePendingMove(final PendingMove p) {
    // move the block
    final DDatanode targetDn = p.target.getDDatanode();
    ExecutorService moveExecutor = targetDn.getMoveExecutor();
    if (moveExecutor == null) {
      final int nThreads = moverThreadAllocator.allocate(maxConcurrentMovesPerNode);
      if (nThreads > 0) {
        moveExecutor = targetDn.initMoveExecutor(nThreads);
    if (moveExecutor == null) {
      LOG.warn("No mover threads available: skip moving " + p);

By default, this causes a pool of 50 threads to be created per target and at some point no
more can be created. If this happens, the pending move is simply skipped.  This causes eternal
hang at {{waitForMoveCompletion()}}, because the pending moves are not removed.  I saw 25
thread pools with 50 threads in the 280-node cluster in this hang state.

IMO, creating more threads is not a scalable solution.  Somehow allowing more thread pool
creation is definitely not a solution either.

> Allow Balancer to run faster
> ----------------------------
>                 Key: HDFS-8818
>                 URL: https://issues.apache.org/jira/browse/HDFS-8818
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer & mover
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>         Attachments: h8818_20150723.patch, h8818_20150727.patch, HDFS-8818-branch-2.7.00.patch
> The original design of Balancer is intentionally to make it run slowly so that the balancing
activities won't affect the normal cluster activities and the running jobs.
> There are new use case that cluster admin may choose to balance the cluster when the
cluster load is low, or in a maintain window.  So that we should have an option to allow Balancer
to run faster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message