hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Prakash <ravihad...@gmail.com>
Subject Re: Lots of Exception for "cannot assign requested address" in datanode logs
Date Wed, 26 Jul 2017 19:56:23 GMT
Hi Omprakash!

DatanodeRegistration happens when the Datanode first hearbeats to the
Namenode. In your case, it seems some other application has acquired the
port 50010 . You can check this with the command "netstat -anp | grep
50010" . Are you trying to run 2 datanode processes on the same machine?

HTH
Ravi

On Wed, Jul 26, 2017 at 5:46 AM, omprakash <omprakashp@cdac.in> wrote:

> Hi all,
>
>
>
> I am running a 4 node cluster with 2 Master node( NN1, NN2 with HA using
> QJM) and 2 Slave nodes(DN1, DN2). I am receiving lots of Exceptions in
> Datanode logs as shown below
>
>
>
> 2017-07-26 17:56:00,703 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> DatanodeRegistration(192.168.9.132:50010, datanodeUuid=5a2e6721-3a9a-43f1-94cc-f58f24b5a15b,
> infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-
> 7aa9fcd4-36fc-4e7b-87cd-d20594774b85;nsid=1753301932;c=1500696043365):Failed
> to transfer BP-1085904515-192.168.9.116-1500696043365:blk_1078544770_4804082
> to 192.168.9.116:50010 got
>
> java.net.BindException: Cannot assign requested address
>
>         at sun.nio.ch.Net.connect0(Native Method)
>
>         at sun.nio.ch.Net.connect(Net.java:465)
>
>         at sun.nio.ch.Net.connect(Net.java:457)
>
>         at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.
> java:670)
>
>         at org.apache.hadoop.net.SocketIOWithTimeout.connect(
> SocketIOWithTimeout.java:192)
>
>         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
>
>         at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
>
>         at org.apache.hadoop.hdfs.server.datanode.DataNode$
> DataTransfer.run(DataNode.java:2312)
>
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
> I have 10 million files in hdfs. All the nodes have same configurations.
> Above Exception started occurring when I changed the below parameters in
> *hdfs-site.xml* file. I made these changes to increase replication rate
> for under-replicated blocks.
>
>
>
> dfs.namenode.handler.count=5000
>
> dfs.namenode.replication.work.multiplier.per.iteration=1000
>
> dfs.namenode.replication.max-streams=2000 *à** not documented in
> hdfs.site.xml*
>
> dfs.namenode.replication.max-streams-hard-limit=4000   *-**à** not
> documented in hdfs.site.xml*
>
>
>
>
>
> The rate of replication of blocks increased but suddenly the Exception
> started to appear.
>
>
>
> Can anybody explain this  behavior?
>
>
>
>
>
> *Regards*
>
> *Omprakash Paliwal*
>
>
>
> ------------------------------------------------------------
> -------------------------------------------------------------------
> [ C-DAC is on Social-Media too. Kindly follow us at:
> Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
>
> This e-mail is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies and the original message. Any unauthorized review, use,
> disclosure, dissemination, forwarding, printing or copying of this email
> is strictly prohibited and appropriate legal action will be taken.
> ------------------------------------------------------------
> -------------------------------------------------------------------
>

Mime
View raw message