Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Thu, 21 Apr 2016 22:48:13 +0000 (UTC)
From: "Tsz Wo Nicholas Sze (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12960079.1461090254000.2509.1461278893073@Atlassian.JIRA>
In-Reply-To: <JIRA.12960079.1461090254000@Atlassian.JIRA>
References: <JIRA.12960079.1461090254000@Atlassian.JIRA>
 <JIRA.12960079.1461090254324@arcas>
Subject: [jira] [Updated] (HDFS-10309) Balancer doesn't honor dfs.blocksize
 value defined with suffix k(kilo), m(mega), g(giga)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-10309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tsz Wo Nicholas Sze updated HDFS-10309:
---------------------------------------
    Summary: Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga)  (was: HDFS Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga))

> Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga)
> ----------------------------------------------------------------------------------------
>
>                 Key: HDFS-10309
>                 URL: https://issues.apache.org/jira/browse/HDFS-10309
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: balancer & mover
>    Affects Versions: 2.8.0
>            Reporter: Amit Anand
>            Assignee: Amit Anand
>            Priority: Minor
>             Fix For: 2.8.0
>
>         Attachments: HDFS-10309.01.patch, HDFS-10309.02.patch
>
>
> While running HDFS Balancer I get error given below when {{dfs.blockSize}} is defined with suffix {{k(kilo), m(mega), g(giga)}} in {{hdfs-site.xml}}. In my deployment {{dfs.blocksize}} is set to {{128m}}. 
> {code}
> hdfs@bcpc-vm1:/home/ubuntu$ hdfs balancer
> 16/04/19 08:49:51 INFO balancer.Balancer: namenodes  = [hdfs://Test-Laptop]
> 16/04/19 08:49:51 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source 
> nodes = 0, #blockpools = 0, run during upgrade = false]
> 16/04/19 08:49:51 INFO balancer.Balancer: included nodes = []
> 16/04/19 08:49:51 INFO balancer.Balancer: excluded nodes = []
> 16/04/19 08:49:51 INFO balancer.Balancer: source nodes = []
> Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved
> 16/04/19 08:49:52 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 16/04/19 08:49:52 INFO block.BlockTokenSecretManager: Setting block keys
> 16/04/19 08:49:52 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
> 16/04/19 08:49:52 INFO block.BlockTokenSecretManager: Setting block keys
> 16/04/19 08:49:52 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)
> Apr 19, 2016 8:49:52 AM  Balancing took 1.408 seconds
> 16/04/19 08:49:52 ERROR balancer.Balancer: Exiting balancer due an exception
> java.lang.NumberFormatException: For input string: "128m"
>         at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>         at java.lang.Long.parseLong(Long.java:589)
>         at java.lang.Long.parseLong(Long.java:631)
>         at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1311)
>         at org.apache.hadoop.hdfs.server.balancer.Balancer.getLong(Balancer.java:221)
>         at org.apache.hadoop.hdfs.server.balancer.Balancer.<init>(Balancer.java:281)
>         at org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:660)
>         at org.apache.hadoop.hdfs.server.balancer.Balancer$Cli.run(Balancer.java:774)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
>         at org.apache.hadoop.hdfs.server.balancer.Balancer.main(Balancer.java:903)
> {code}
> However, the workaround for this is to run {{hdfs balancer}} with passing numeric value for {{dfs.blocksize}} or change your {{hdfs-site.xml}}.
> {code}
> hdfs balancer -Ddfs.blocksize=134217728
> {code}


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)