Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3F40010EC7 for ; Thu, 20 Feb 2014 11:17:04 +0000 (UTC) Received: (qmail 57977 invoked by uid 500); 20 Feb 2014 11:16:48 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 57813 invoked by uid 500); 20 Feb 2014 11:16:44 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 57765 invoked by uid 99); 20 Feb 2014 11:16:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Feb 2014 11:16:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of praveenesh@gmail.com designates 209.85.219.46 as permitted sender) Received: from [209.85.219.46] (HELO mail-oa0-f46.google.com) (209.85.219.46) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Feb 2014 11:16:34 +0000 Received: by mail-oa0-f46.google.com with SMTP id n16so1946034oag.33 for ; Thu, 20 Feb 2014 03:16:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=1YnC78Y6aCUw/enHayiJtXHi390n6Oc99/6XpmioLko=; b=jKdsaYJGyEWCX/7v7TqojPZONrajNt1aeI1eYTT4RcVasRJzK6hIL0jI0407vrK4qq yMD+W2fBzxGYOXN23wtbqG1Z6Qe3YfSeZgjPFbpt/y8oJY67vqJhXrgCQ5mr1LiJO7Xr gbTigqj83lYcbU0djqScHL45DoiqqQiuGIBT3jn2yZehNqaIE5E9FKyPY2Ww9w5c+5cu 2VweZf6U+D+KNpz89+junvw9UrkVyakobU2lBi96NTnRmw3gskRmXSANwaK1eDqYrhgm zvqGB8DhSuKYAMw79nVCjBctth+OsT2+ADbIPimr3GNNiOHW6MlBfAXxKAgch4TjgTR+ onaA== MIME-Version: 1.0 X-Received: by 10.60.45.206 with SMTP id p14mr910094oem.21.1392894972536; Thu, 20 Feb 2014 03:16:12 -0800 (PST) Received: by 10.182.95.6 with HTTP; Thu, 20 Feb 2014 03:16:12 -0800 (PST) In-Reply-To: References: Date: Thu, 20 Feb 2014 11:16:12 +0000 Message-ID: Subject: Re: A question about Balancer in HDFS From: praveenesh kumar To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=001a11c2019a0c2c3604f2d4a2de X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2019a0c2c3604f2d4a2de Content-Type: text/plain; charset=ISO-8859-1 Have you noticed any balancing in your datanodes by looking at the namenode url (where it shows data-node level usage). You should have. I think its not stuck, it just waiting for looking for more data blocks which it can move around to balance the cluster. There is threshold limit (if I remember it correctly), which you can also set while running the balancer (otherwise it uses some default value) and keeps on running till it reaches that threshold. I think it would be safe to just stop the process, as it has done balancing. Sorry I don't have the exact answer, just sharing my experiences. Someone else might throw better light in here. Regards Prav On Thu, Feb 20, 2014 at 7:09 AM, QShu wrote: > Hi, Guys: > > The HDFS Balancer will stuck after running a while in my cluster > now, the following is the log, after 14:38:52, the Balancer stuccoed and > never continue balancing. > Can anybody give me some help, thank you! > > > ==============================================log============================================== > 14/02/20 14:38:25 INFO balancer.Balancer: namenodes = [hdfs://hstore] > 14/02/20 14:38:25 INFO balancer.Balancer: p = > Balancer.Parameters[BalancingPolicy.Node, threshold=10.0] > Time Stamp Iteration# Bytes Already Moved Bytes Left To > Move Bytes Being Moved > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.111:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.103:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.108:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.102:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/ > 172.19.105.23:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.112:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.106:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.101:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/ > 172.19.105.28:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.113:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.110:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/ > 172.19.105.27:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.116:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.105:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/ > 172.19.105.29:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/ > 172.19.105.25:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/ > 172.19.105.26:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/ > 172.19.105.24:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.114:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.107:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.109:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/ > 172.19.105.30:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.115:50010 > 14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/ > 172.19.105.104:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[ > 172.19.105.102:50010, utilization=42.58400812492816], Source[ > 172.19.105.23:50010, utilization=41.94449751804609], Source[ > 172.19.105.101:50010, utilization=49.25646255747681], Source[ > 172.19.105.27:50010, utilization=42.13637544454659], Source[ > 172.19.105.29:50010, utilization=42.253685824453484], Source[ > 172.19.105.25:50010, utilization=41.29288653593026], Source[ > 172.19.105.26:50010, utilization=42.58689306059978], Source[ > 172.19.105.24:50010, utilization=42.681174402528306], Source[ > 172.19.105.30:50010, utilization=42.111934753109566]] > 14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized: > [BalancerDatanode[172.19.105.28:50010, utilization=5.073501546034823]] > 14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the > cluster balanced. > 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from > 172.19.105.102:50010 to 172.19.105.28:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from > 172.19.105.23:50010 to 172.19.105.111:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from > 172.19.105.101:50010 to 172.19.105.103:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from > 172.19.105.27:50010 to 172.19.105.108:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from > 172.19.105.29:50010 to 172.19.105.112:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from > 172.19.105.25:50010 to 172.19.105.106:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from > 172.19.105.26:50010 to 172.19.105.113:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from > 172.19.105.24:50010 to 172.19.105.110:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from > 172.19.105.30:50010 to 172.19.105.116:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration > 2014-2-20 14:38:26 0 0 B 1.23 > TB 90 GB > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074013803 with a length of 128 MB bytes from 172.19.105.26:50010 to > 172.19.105.113:50010 using proxy source 172.19.105.102:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073936074 with a length of 128 MB bytes from 172.19.105.101:50010 to > 172.19.105.103:50010 using proxy source 172.19.105.101:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073961733 with a length of 128 MB bytes from 172.19.105.30:50010 to > 172.19.105.116:50010 using proxy source 172.19.105.101:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073922224 with a length of 128 MB bytes from 172.19.105.23:50010 to > 172.19.105.111:50010 using proxy source 172.19.105.102:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074018599 with a length of 26.12 MB bytes from 172.19.105.27:50010 to > 172.19.105.108:50010 using proxy source 172.19.105.102:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074013794 with a length of 128 MB bytes from 172.19.105.26:50010 to > 172.19.105.113:50010 using proxy source 172.19.105.101:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from > 172.19.105.102:50010 to 172.19.105.113:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074013798 with a length of 128 MB bytes from 172.19.105.26:50010 to > 172.19.105.113:50010 using proxy source 172.19.105.101:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from > 172.19.105.101:50010 to 172.19.105.113:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074018603 with a length of 128 MB bytes from 172.19.105.27:50010 to > 172.19.105.108:50010 using proxy source 172.19.105.102:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from > 172.19.105.102:50010 to 172.19.105.108:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074008476 with a length of 128 MB bytes from 172.19.105.25:50010 to > 172.19.105.106:50010 using proxy source 172.19.105.107:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073922231 with a length of 128 MB bytes from 172.19.105.23:50010 to > 172.19.105.111:50010 using proxy source 172.19.105.114:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from > 172.19.105.102:50010 to 172.19.105.111:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073812199 with a length of 64.75 MB bytes from 172.19.105.102:50010 to > 172.19.105.28:50010 using proxy source 172.19.105.24:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073991855 with a length of 73.88 MB bytes from 172.19.105.24:50010 to > 172.19.105.110:50010 using proxy source 172.19.105.102:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074035591 with a length of 128 MB bytes from 172.19.105.29:50010 to > 172.19.105.112:50010 using proxy source 172.19.105.101:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074008479 with a length of 128 MB bytes from 172.19.105.25:50010 to > 172.19.105.106:50010 using proxy source 172.19.105.107:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from > 172.19.105.107:50010 to 172.19.105.106:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074035589 with a length of 6.51 KB bytes from 172.19.105.29:50010 to > 172.19.105.112:50010 using proxy source 172.19.105.103:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from > 172.19.105.101:50010 to 172.19.105.112:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073997260 with a length of 85.24 KB bytes from 172.19.105.101:50010 to > 172.19.105.103:50010 using proxy source 172.19.105.106:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073992320 with a length of 128 MB bytes from 172.19.105.30:50010 to > 172.19.105.116:50010 using proxy source 172.19.105.105:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from > 172.19.105.101:50010 to 172.19.105.116:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from > 172.19.105.101:50010 to 172.19.105.103:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074009436 with a length of 128 MB bytes from 172.19.105.101:50010 to > 172.19.105.103:50010 using proxy source 172.19.105.28:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from > 172.19.105.106:50010 to 172.19.105.103:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from > 172.19.105.102:50010 to 172.19.105.110:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074009439 with a length of 128 MB bytes from 172.19.105.101:50010 to > 172.19.105.103:50010 using proxy source 172.19.105.27:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from > 172.19.105.28:50010 to 172.19.105.103:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from > 172.19.105.24:50010 to 172.19.105.28:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073883114 with a length of 128 MB bytes from 172.19.105.102:50010 to > 172.19.105.28:50010 using proxy source 172.19.105.27:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074018604 with a length of 128 MB bytes from 172.19.105.27:50010 to > 172.19.105.108:50010 using proxy source 172.19.105.24:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from > 172.19.105.102:50010 to 172.19.105.108:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074013709 with a length of 128 MB bytes from 172.19.105.26:50010 to > 172.19.105.113:50010 using proxy source 172.19.105.27:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from > 172.19.105.101:50010 to 172.19.105.113:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from > 172.19.105.27:50010 to 172.19.105.103:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073987402 with a length of 75.99 MB bytes from 172.19.105.24:50010 to > 172.19.105.110:50010 using proxy source 172.19.105.116:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073987410 with a length of 128 MB bytes from 172.19.105.24:50010 to > 172.19.105.110:50010 using proxy source 172.19.105.24:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from > 172.19.105.116:50010 to 172.19.105.110:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073987422 with a length of 128 MB bytes from 172.19.105.24:50010 to > 172.19.105.110:50010 using proxy source 172.19.105.30:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073922228 with a length of 128 MB bytes from 172.19.105.23:50010 to > 172.19.105.111:50010 using proxy source 172.19.105.23:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from > 172.19.105.24:50010 to 172.19.105.110:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from > 172.19.105.114:50010 to 172.19.105.111:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073922442 with a length of 128 MB bytes from 172.19.105.30:50010 to > 172.19.105.116:50010 using proxy source 172.19.105.26:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from > 172.19.105.105:50010 to 172.19.105.116:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073992340 with a length of 128 MB bytes from 172.19.105.30:50010 to > 172.19.105.116:50010 using proxy source 172.19.105.111:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from > 172.19.105.26:50010 to 172.19.105.116:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from > 172.19.105.111:50010 to 172.19.105.116:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074008469 with a length of 128 MB bytes from 172.19.105.25:50010 to > 172.19.105.106:50010 using proxy source 172.19.105.107:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from > 172.19.105.107:50010 to 172.19.105.106:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074035711 with a length of 128 MB bytes from 172.19.105.29:50010 to > 172.19.105.112:50010 using proxy source 172.19.105.29:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from > 172.19.105.103:50010 to 172.19.105.112:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073992638 with a length of 128 MB bytes from 172.19.105.23:50010 to > 172.19.105.111:50010 using proxy source 172.19.105.23:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from > 172.19.105.23:50010 to 172.19.105.111:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073917025 with a length of 128 MB bytes from 172.19.105.24:50010 to > 172.19.105.110:50010 using proxy source 172.19.105.26:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073892376 with a length of 128 MB bytes from 172.19.105.26:50010 to > 172.19.105.113:50010 using proxy source 172.19.105.106:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from > 172.19.105.27:50010 to 172.19.105.28:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073917544 with a length of 128 MB bytes from 172.19.105.102:50010 to > 172.19.105.28:50010 using proxy source 172.19.105.108:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073990086 with a length of 128 MB bytes from 172.19.105.27:50010 to > 172.19.105.108:50010 using proxy source 172.19.105.27:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from > 172.19.105.30:50010 to 172.19.105.110:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from > 172.19.105.24:50010 to 172.19.105.108:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from > 172.19.105.27:50010 to 172.19.105.113:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from > 172.19.105.23:50010 to 172.19.105.111:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074035705 with a length of 128 MB bytes from 172.19.105.29:50010 to > 172.19.105.112:50010 using proxy source 172.19.105.23:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from > 172.19.105.29:50010 to 172.19.105.112:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from > 172.19.105.107:50010 to 172.19.105.106:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from > 172.19.105.108:50010 to 172.19.105.28:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from > 172.19.105.27:50010 to 172.19.105.108:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1073911911 with a length of 128 MB bytes from 172.19.105.29:50010 to > 172.19.105.112:50010 using proxy source 172.19.105.29:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from > 172.19.105.26:50010 to 172.19.105.110:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from > 172.19.105.106:50010 to 172.19.105.113:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from > 172.19.105.23:50010 to 172.19.105.112:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from > 172.19.105.29:50010 to 172.19.105.112:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from > 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.103:50010is succeeded. > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074035693 with a length of 128 MB bytes from 172.19.105.29:50010 to > 172.19.105.112:50010 using proxy source 172.19.105.29:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074009432 with a length of 128 MB bytes from 172.19.105.101:50010 to > 172.19.105.103:50010 using proxy source 172.19.105.29:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from > 172.19.105.29:50010 to 172.19.105.112:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from > 172.19.105.29:50010 to 172.19.105.103:50010 > 14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from > 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.106:50010is succeeded. > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074009435 with a length of 128 MB bytes from 172.19.105.101:50010 to > 172.19.105.103:50010 using proxy source 172.19.105.27:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block > 1074008471 with a length of 128 MB bytes from 172.19.105.25:50010 to > 172.19.105.106:50010 using proxy source 172.19.105.107:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from > 172.19.105.27:50010 to 172.19.105.103:50010 > 14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from > 172.19.105.107:50010 to 172.19.105.106:50010 > 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from > 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded. > 14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from > 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.24:50010is succeeded. > 14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from > 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.116:50010is succeeded. > 14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from > 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.114:50010is succeeded. > 14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from > 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.108:50010is succeeded. > 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from > 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.102:50010is succeeded. > 14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from > 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.102:50010is succeeded. > 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from > 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.101:50010is succeeded. > 14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from > 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.105:50010is succeeded. > 14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from > 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.101:50010is succeeded. > 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from > 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.102:50010is succeeded. > 14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from > 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded. > 14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from > 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.28:50010is succeeded. > 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from > 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded. > 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from > 172.19.105.102:50010 to 172.19.105.28:50010 through 172.19.105.27:50010is succeeded. > 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from > 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded. > 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from > 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.27:50010is succeeded. > 14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from > 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded. > 14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from > 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.30:50010is succeeded. > 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from > 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded. > 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from > 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.29:50010is succeeded. > 14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from > 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.101:50010is succeeded. > 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from > 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded. > 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from > 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.27:50010is succeeded. > 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from > 172.19.105.25:50010 to 172.19.105.106:50010 through 172.19.105.107:50010is succeeded. > 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from > 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.111:50010is succeeded. > 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from > 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.24:50010is succeeded. > 14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from > 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.26:50010is succeeded. > 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from > 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded. > 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from > 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.101:50010is succeeded. > 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from > 172.19.105.27:50010 to 172.19.105.108:50010 through 172.19.105.102:50010is succeeded. > 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from > 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.29:50010is succeeded. > 14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from > 172.19.105.30:50010 to 172.19.105.116:50010 through 172.19.105.26:50010is succeeded. > 14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from > 172.19.105.26:50010 to 172.19.105.113:50010 through 172.19.105.106:50010is succeeded. > 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from > 172.19.105.24:50010 to 172.19.105.110:50010 through 172.19.105.24:50010is succeeded. > 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from > 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded. > 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from > 172.19.105.23:50010 to 172.19.105.111:50010 through 172.19.105.23:50010is succeeded. > 14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from > 172.19.105.101:50010 to 172.19.105.103:50010 through 172.19.105.27:50010is succeeded. > 14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from > 172.19.105.29:50010 to 172.19.105.112:50010 through 172.19.105.23:50010is succeeded. --001a11c2019a0c2c3604f2d4a2de Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Have you noticed any balancing in your data= nodes by looking at the namenode url (where it shows data-node level usage)= . You should have.
I think its not stuck, it just waiting for looking fo= r more data blocks which it can move around to balance the cluster.
There is threshold limit (if I remember it correctly), which you can = also set while running the balancer (otherwise it uses some default value) = and keeps on running till it reaches that threshold. I think it would be sa= fe to just stop the process, as it has done balancing.

Sorry I don't have the exact answer, just sharing my experiences. S= omeone else might throw better light in here.

Regards
Prav


On Thu, Feb 20, 2014 at 7:09 AM, QShu <shuyunkun@gmail.com> wrote:
Hi, Guys:

=A0 =A0 =A0 =A0 The HDFS Balancer will stuck after running a while in my cl= uster now, the following is the log, after 14:38:52, the Balancer stuccoed = and never continue balancing.
=A0 =A0 =A0 =A0 Can anybody give me some help, thank you!

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3Dlog=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
14/02/20 14:38:25 INFO balancer.Balancer: namenodes =3D [hdfs://hstore]
14/02/20 14:38:25 INFO balancer.Balancer: p =A0 =A0 =A0 =A0 =3D Balancer.Pa= rameters[BalancingPolicy.Node, threshold=3D10.0]
Time Stamp =A0 =A0 =A0 =A0 =A0 =A0 =A0 Iteration# =A0Bytes Already Moved = =A0Bytes Left To Move =A0Bytes Being Moved
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.111:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.103:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.108:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.102:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.23:50010=
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.112:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.106:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.101:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.28:50010=
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.113:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.110:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.27:50010=
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.116:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.105:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.29:50010=
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.25:50010=
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.26:50010=
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.24:50010=
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.114:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.107:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.109:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack1/172.19.105.30:50010=
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.115:50010
14/02/20 14:38:26 INFO net.NetworkTopology: Adding a new node: /rack2/172.19.105.104:50010
14/02/20 14:38:26 INFO balancer.Balancer: 9 over-utilized: [Source[172.19.105.102:50010= , utilization=3D42.58400812492816], Source[172.19.105.23:50010, utilization=3D41.94449751= 804609], Source[1= 72.19.105.101:50010, utilization=3D49.25646255747681], Source[172.19.105.27:50010, = utilization=3D42.13637544454659], Source[172.19.105.29:50010, utilization=3D42.2536858244= 53484], Source[172= .19.105.25:50010, utilization=3D41.29288653593026], Source[172.19.105.26:50010, utili= zation=3D42.58689306059978], Source[172.19.105.24:50010, utilization=3D42.681174402528306= ], Source[172.19.1= 05.30:50010, utilization=3D42.111934753109566]]
14/02/20 14:38:26 INFO balancer.Balancer: 1 underutilized: [BalancerDatanod= e[172.19.105.28:50= 010, utilization=3D5.073501546034823]]
14/02/20 14:38:26 INFO balancer.Balancer: Need to move 1.23 TB to make the = cluster balanced.
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from = 172.19.105.102:50= 010 to 172.19.= 105.28:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from = 172.19.105.23:5001= 0 to 172.19.1= 05.111:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from = 172.19.105.101:50= 010 to 172.19= .105.103:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from = 172.19.105.27:5001= 0 to 172.19.1= 05.108:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from = 172.19.105.29:5001= 0 to 172.19.1= 05.112:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from = 172.19.105.25:5001= 0 to 172.19.1= 05.106:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from = 172.19.105.26:5001= 0 to 172.19.1= 05.113:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from = 172.19.105.24:5001= 0 to 172.19.1= 05.110:50010
14/02/20 14:38:26 INFO balancer.Balancer: Decided to move 10 GB bytes from = 172.19.105.30:5001= 0 to 172.19.1= 05.116:50010
14/02/20 14:38:26 INFO balancer.Balancer: Will move 90 GB in this iteration=
2014-2-20 14:38:26 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A00 B =A0 =A0 =A0 =A0 =A0 =A0 1.23 TB =A0 =A0 =A0 =A0 =A0 =A0 =A0= 90 GB
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013803= with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.102:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073936074= with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source <= a href=3D"http://172.19.105.101:50010" target=3D"_blank">172.19.105.101:500= 10
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073961733= with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.101:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922224= with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.102:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018599= with a length of 26.12 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source <= a href=3D"http://172.19.105.102:50010" target=3D"_blank">172.19.105.102:500= 10
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013794= with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013803 from = 172.19.105.102:50= 010 to 172.19= .105.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013798= with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.101:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013794 from = 172.19.105.101:50= 010 to 172.19= .105.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018603= with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.102:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018599 from = 172.19.105.102:50= 010 to 172.19= .105.108:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008476= with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922231= with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.114:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922224 from = 172.19.105.102:50= 010 to 172.19= .105.111:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073812199= with a length of 64.75 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source <= a href=3D"http://172.19.105.24:50010" target=3D"_blank">172.19.105.24:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073991855= with a length of 73.88 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source <= a href=3D"http://172.19.105.102:50010" target=3D"_blank">172.19.105.102:500= 10
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035591= with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.101:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008479= with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008476 from = 172.19.105.107:50= 010 to 172.19= .105.106:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035589= with a length of 6.51 KB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.103:5001= 0
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035591 from = 172.19.105.101:50= 010 to 172.19= .105.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073997260= with a length of 85.24 KB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source= 172.19.105.106:5= 0010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992320= with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.105:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073961733 from = 172.19.105.101:50= 010 to 172.19= .105.116:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073936074 from = 172.19.105.101:50= 010 to 172.19= .105.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009436= with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source <= a href=3D"http://172.19.105.28:50010" target=3D"_blank">172.19.105.28:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073997260 from = 172.19.105.106:50= 010 to 172.19= .105.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073991855 from = 172.19.105.102:50= 010 to 172.19= .105.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009439= with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source <= a href=3D"http://172.19.105.27:50010" target=3D"_blank">172.19.105.27:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009436 from = 172.19.105.28:5001= 0 to 172.19.1= 05.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073812199 from = 172.19.105.24:5001= 0 to 172.19.10= 5.28:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073883114= with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.27:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074018604= with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.24:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018603 from = 172.19.105.102:50= 010 to 172.19= .105.108:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074013709= with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.27:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013798 from = 172.19.105.101:50= 010 to 172.19= .105.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009439 from = 172.19.105.27:5001= 0 to 172.19.1= 05.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987402= with a length of 75.99 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source <= a href=3D"http://172.19.105.116:50010" target=3D"_blank">172.19.105.116:500= 10
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987410= with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.24:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987402 from = 172.19.105.116:50= 010 to 172.19= .105.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073987422= with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.30:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922228= with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987410 from = 172.19.105.24:5001= 0 to 172.19.1= 05.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922231 from = 172.19.105.114:50= 010 to 172.19= .105.111:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073922442= with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.26:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992320 from = 172.19.105.105:50= 010 to 172.19= .105.116:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992340= with a length of 128 MB bytes from 172.19.105.30:50010 to 172.19.105.116:50010 using proxy source 172.19.105.111:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922442 from = 172.19.105.26:5001= 0 to 172.19.1= 05.116:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992340 from = 172.19.105.111:50= 010 to 172.19= .105.116:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008469= with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008479 from = 172.19.105.107:50= 010 to 172.19= .105.106:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035711= with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035589 from = 172.19.105.103:50= 010 to 172.19= .105.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073992638= with a length of 128 MB bytes from 172.19.105.23:50010 to 172.19.105.111:50010 using proxy source 172.19.105.23:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073922228 from = 172.19.105.23:5001= 0 to 172.19.1= 05.111:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917025= with a length of 128 MB bytes from 172.19.105.24:50010 to 172.19.105.110:50010 using proxy source 172.19.105.26:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073892376= with a length of 128 MB bytes from 172.19.105.26:50010 to 172.19.105.113:50010 using proxy source 172.19.105.106:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073883114 from = 172.19.105.27:5001= 0 to 172.19.10= 5.28:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073917544= with a length of 128 MB bytes from 172.19.105.102:50010 to 172.19.105.28:50010 using proxy source 172.19.105.108:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073990086= with a length of 128 MB bytes from 172.19.105.27:50010 to 172.19.105.108:50010 using proxy source 172.19.105.27:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073987422 from = 172.19.105.30:5001= 0 to 172.19.1= 05.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074018604 from = 172.19.105.24:5001= 0 to 172.19.1= 05.108:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074013709 from = 172.19.105.27:5001= 0 to 172.19.1= 05.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073992638 from = 172.19.105.23:5001= 0 to 172.19.1= 05.111:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035705= with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.23:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035711 from = 172.19.105.29:5001= 0 to 172.19.1= 05.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008469 from = 172.19.105.107:50= 010 to 172.19= .105.106:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917544 from = 172.19.105.108:50= 010 to 172.19.= 105.28:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073990086 from = 172.19.105.27:5001= 0 to 172.19.1= 05.108:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1073911911= with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073917025 from = 172.19.105.26:5001= 0 to 172.19.1= 05.110:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073892376 from = 172.19.105.106:50= 010 to 172.19= .105.113:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035705 from = 172.19.105.23:5001= 0 to 172.19.1= 05.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1073911911 from = 172.19.105.29:5001= 0 to 172.19.1= 05.112:50010
14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1074035589 from 172.19.105.29:50010 to 172.19.105.1= 12:50010 through 172.19.105.103:50010 is succeeded.
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074035693= with a length of 128 MB bytes from 172.19.105.29:50010 to 172.19.105.112:50010 using proxy source 172.19.105.29:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009432= with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source <= a href=3D"http://172.19.105.29:50010" target=3D"_blank">172.19.105.29:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074035693 from = 172.19.105.29:5001= 0 to 172.19.1= 05.112:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009432 from = 172.19.105.29:5001= 0 to 172.19.1= 05.103:50010
14/02/20 14:38:26 INFO balancer.Balancer: Moving block 1073997260 from 172.19.105.101:50010<= /a> to 172.19.105= .103:50010 through 172.19.105.106:50010 is succeeded.
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074009435= with a length of 128 MB bytes from 172.19.105.101:50010 to 172.19.105.103:50010 using proxy source <= a href=3D"http://172.19.105.27:50010" target=3D"_blank">172.19.105.27:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Decided to move block 1074008471= with a length of 128 MB bytes from 172.19.105.25:50010 to 172.19.105.106:50010 using proxy source 172.19.105.107:50010=
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074009435 from = 172.19.105.27:5001= 0 to 172.19.1= 05.103:50010
14/02/20 14:38:26 DEBUG balancer.Balancer: Starting moving 1074008471 from = 172.19.105.107:50= 010 to 172.19= .105.106:50010
14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1074018599 from 172.19.105.27:50010 to 172.19.105.1= 08:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:32 INFO balancer.Balancer: Moving block 1073812199 from 172.19.105.102:50010<= /a> to 172.19.105.= 28:50010 through 172.19.105.24:50010 is succeeded.
14/02/20 14:38:33 INFO balancer.Balancer: Moving block 1073987402 from 172.19.105.24:50010 to 172.19.105.1= 10:50010 through 172.19.105.116:50010 is succeeded.
14/02/20 14:38:37 INFO balancer.Balancer: Moving block 1073922231 from 172.19.105.23:50010 to 172.19.105.1= 11:50010 through 172.19.105.114:50010 is succeeded.
14/02/20 14:38:38 INFO balancer.Balancer: Moving block 1073917544 from 172.19.105.102:50010<= /a> to 172.19.105.= 28:50010 through 172.19.105.108:50010 is succeeded.
14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1073922224 from 172.19.105.23:50010 to 172.19.105.1= 11:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:39 INFO balancer.Balancer: Moving block 1074013803 from 172.19.105.26:50010 to 172.19.105.1= 13:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1074035591 from 172.19.105.29:50010 to 172.19.105.1= 12:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:40 INFO balancer.Balancer: Moving block 1073992320 from 172.19.105.30:50010 to 172.19.105.1= 16:50010 through 172.19.105.105:50010 is succeeded.
14/02/20 14:38:41 INFO balancer.Balancer: Moving block 1073961733 from 172.19.105.30:50010 to 172.19.105.1= 16:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1073991855 from 172.19.105.24:50010 to 172.19.105.1= 10:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:42 INFO balancer.Balancer: Moving block 1074008479 from 172.19.105.25:50010 to 172.19.105.1= 06:50010 through 172.19.105.107:50010 is succeeded.
14/02/20 14:38:43 INFO balancer.Balancer: Moving block 1074009436 from 172.19.105.101:50010<= /a> to 172.19.105= .103:50010 through 172.19.105.28:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008469 from 172.19.105.25:50010 to 172.19.105.1= 06:50010 through 172.19.105.107:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073883114 from 172.19.105.102:50010<= /a> to 172.19.105.= 28:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073911911 from 172.19.105.29:50010 to 172.19.105.1= 12:50010 through 172.19.105.29:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1073990086 from 172.19.105.27:50010 to 172.19.105.1= 08:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:44 INFO balancer.Balancer: Moving block 1074008476 from 172.19.105.25:50010 to 172.19.105.1= 06:50010 through 172.19.105.107:50010 is succeeded.
14/02/20 14:38:45 INFO balancer.Balancer: Moving block 1073987422 from 172.19.105.24:50010 to 172.19.105.1= 10:50010 through 172.19.105.30:50010 is succeeded.
14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074013794 from 172.19.105.26:50010 to 172.19.105.1= 13:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1074009432 from 172.19.105.101:50010<= /a> to 172.19.105= .103:50010 through 172.19.105.29:50010 is succeeded.
14/02/20 14:38:46 INFO balancer.Balancer: Moving block 1073936074 from 172.19.105.101:50010<= /a> to 172.19.105= .103:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074035711 from 172.19.105.29:50010 to 172.19.105.1= 12:50010 through 172.19.105.29:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074013709 from 172.19.105.26:50010 to 172.19.105.1= 13:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074008471 from 172.19.105.25:50010 to 172.19.105.1= 06:50010 through 172.19.105.107:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073992340 from 172.19.105.30:50010 to 172.19.105.1= 16:50010 through 172.19.105.111:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1074018604 from 172.19.105.27:50010 to 172.19.105.1= 08:50010 through 172.19.105.24:50010 is succeeded.
14/02/20 14:38:47 INFO balancer.Balancer: Moving block 1073917025 from 172.19.105.24:50010 to 172.19.105.1= 10:50010 through 172.19.105.26:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074009439 from 172.19.105.101:50010<= /a> to 172.19.105= .103:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074013798 from 172.19.105.26:50010 to 172.19.105.1= 13:50010 through 172.19.105.101:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074018603 from 172.19.105.27:50010 to 172.19.105.1= 08:50010 through 172.19.105.102:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1074035693 from 172.19.105.29:50010 to 172.19.105.1= 12:50010 through 172.19.105.29:50010 is succeeded.
14/02/20 14:38:48 INFO balancer.Balancer: Moving block 1073922442 from 172.19.105.30:50010 to 172.19.105.1= 16:50010 through 172.19.105.26:50010 is succeeded.
14/02/20 14:38:49 INFO balancer.Balancer: Moving block 1073892376 from 172.19.105.26:50010 to 172.19.105.1= 13:50010 through 172.19.105.106:50010 is succeeded.
14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073987410 from 172.19.105.24:50010 to 172.19.105.1= 10:50010 through 172.19.105.24:50010 is succeeded.
14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073922228 from 172.19.105.23:50010 to 172.19.105.1= 11:50010 through 172.19.105.23:50010 is succeeded.
14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1073992638 from 172.19.105.23:50010 to 172.19.105.1= 11:50010 through 172.19.105.23:50010 is succeeded.
14/02/20 14:38:50 INFO balancer.Balancer: Moving block 1074009435 from 172.19.105.101:50010<= /a> to 172.19.105= .103:50010 through 172.19.105.27:50010 is succeeded.
14/02/20 14:38:52 INFO balancer.Balancer: Moving block 1074035705 from 172.19.105.29:50010 to 172.19.105.1= 12:50010 through 172.19.105.23:50010 is succeeded.

--001a11c2019a0c2c3604f2d4a2de--