Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
MIME-Version: 1.0
Reply-To: ashwanthkumar@googlemail.com
Sender: ashwanth.kumar@gmail.com
From: Ashwanth Kumar <ashwanthkumar@googlemail.com>
Date: Sun, 15 Nov 2015 09:24:57 +0530
Message-ID: 
 <CAD9m7CwPvW5FJ5jV+MwA_Xawsoqa-n_Hs4GQFtu-4SeAg9rGfQ@mail.gmail.com>
Subject: Unable to submit jobs to a Hadoop cluster after a while
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e0111bea2ed9d2705248c4375

--089e0111bea2ed9d2705248c4375
Content-Type: text/plain; charset=UTF-8

We're running Hadoop 2.6.0 via CDH5.4.4 and we get the following error
while submitting a new job

15/10/08 00:33:31 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/data/hadoopfs/mapred/staging/hadoop/.staging/job_201510050004_0388/job.jar
could only be replicated to 0 nodes instead of minReplication (=1).  There
are 161 datanode(s) running and no node(s) are excluded in this operation.

At that time we had 161 DNs running in the cluster. From the NN logs I see

2015-10-08 01:00:26,889 DEBUG
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed
to choose remote rack (location = ~/default-rack), fallback to local rack
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:691)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:580)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:357)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:419)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:214)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:111)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.chooseTargets(BlockManager.java:3746)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.access$200(BlockManager.java:3711)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1400)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1306)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3682)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3634)
at java.lang.Thread.run(Thread.java:722)
2015-10-08 01:00:26,890 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed
to place enough replicas, still in need of 1 to reach 3
(unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7,
storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
newBlock=false) [

>From one of the live 160+ DN logs, we saw

Node /default-rack/10.181.8.222:50010 [
  Storage [DISK]DS-2d39f3c3-2e67-48ad-871b-632f66b277d7:NORMAL:
10.181.8.222:50010 is not chosen since the node is too busy (load: 2 >
1.8370786516853932) .
]
Node /default-rack/10.181.25.147:50010 [
  Storage [DISK]DS-60b511b0-62aa-4c0f-92d9-6d90ff32ee49:NORMAL:
10.181.25.147:50010 is not chosen since the node is too busy (load: 2 >
1.8370786516853932) .
]
Node /default-rack/10.181.8.152:50010 [
  Storage [DISK]DS-7e0bf761-86f2-4748-9eda-fbfd9c69e127:NORMAL:
10.181.8.152:50010 is not chosen since the node is too busy (load: 2 >
1.8370786516853932) .
]
Node /default-rack/10.181.25.67:50010 [
  Storage [DISK]DS-5849e4d8-4ab6-4392-aee2-7a354c82c19d:NORMAL:
10.181.25.67:50010 is not chosen since the node is too busy (load: 2 >
1.8370786516853932) .
]


Few things we observed from our end
- If we restart the NN, we're able to submit jobs without any issues
- We run this Hadoop cluster on AWS
- DN and TT process run on a single EC2 machine which is backed by an
AutoScaling Group.
- We've another cluster which does't autoscale and doesn't exhibit the
behaviour

Any pointers or ideas on how to solve this for good would be really
appreciated.

-- 

Ashwanth Kumar / ashwanthkumar.in

--089e0111bea2ed9d2705248c4375
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">We&#39;re running Hadoop 2.6.0 via CDH5.4.4 and we get the=
 following error while submitting a new job<div><br></div><div><div>15/10/0=
8 00:33:31 WARN security.UserGroupInformation: PriviledgedActionException a=
s:hadoop (auth:SIMPLE) cause:org.apache.hadoop.ipc.RemoteException(java.io.=
IOException): File /data/hadoopfs/mapred/staging/hadoop/.staging/job_201510=
050004_0388/job.jar could only be replicated to 0 nodes instead of minRepli=
cation (=3D1).=C2=A0 There are 161 datanode(s) running and no node(s) are e=
xcluded in this operation.</div><div><br></div><div>At that time we had 161=
 DNs running in the cluster. From the NN logs I see</div><div><br></div><di=
v><div>2015-10-08 01:00:26,889 DEBUG org.apache.hadoop.hdfs.server.blockman=
agement.BlockPlacementPolicy: Failed to choose remote rack (location =3D ~/=
default-rack), fallback to local rack</div><div>org.apache.hadoop.hdfs.serv=
er.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:=C2=A0</=
div><div><span class=3D"" style=3D"white-space:pre">	</span>at org.apache.h=
adoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(=
BlockPlacementPolicyDefault.java:691)</div><div><span class=3D"" style=3D"w=
hite-space:pre">	</span>at org.apache.hadoop.hdfs.server.blockmanagement.Bl=
ockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java=
:580)</div><div><span class=3D"" style=3D"white-space:pre">	</span>at org.a=
pache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.choose=
Target(BlockPlacementPolicyDefault.java:357)</div><div><span class=3D"" sty=
le=3D"white-space:pre">	</span>at org.apache.hadoop.hdfs.server.blockmanage=
ment.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.j=
ava:419)</div><div><span class=3D"" style=3D"white-space:pre">	</span>at or=
g.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.cho=
oseTarget(BlockPlacementPolicyDefault.java:214)</div><div><span class=3D"" =
style=3D"white-space:pre">	</span>at org.apache.hadoop.hdfs.server.blockman=
agement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefaul=
t.java:111)</div><div><span class=3D"" style=3D"white-space:pre">	</span>at=
 org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork=
.chooseTargets(BlockManager.java:3746)</div><div><span class=3D"" style=3D"=
white-space:pre">	</span>at org.apache.hadoop.hdfs.server.blockmanagement.B=
lockManager$ReplicationWork.access$200(BlockManager.java:3711)</div><div><s=
pan class=3D"" style=3D"white-space:pre">	</span>at org.apache.hadoop.hdfs.=
server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockMa=
nager.java:1400)</div><div><span class=3D"" style=3D"white-space:pre">	</sp=
an>at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeRep=
licationWork(BlockManager.java:1306)</div><div><span class=3D"" style=3D"wh=
ite-space:pre">	</span>at org.apache.hadoop.hdfs.server.blockmanagement.Blo=
ckManager.computeDatanodeWork(BlockManager.java:3682)</div><div><span class=
=3D"" style=3D"white-space:pre">	</span>at org.apache.hadoop.hdfs.server.bl=
ockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3634)</=
div><div><span class=3D"" style=3D"white-space:pre">	</span>at java.lang.Th=
read.run(Thread.java:722)</div><div>2015-10-08 01:00:26,890 WARN org.apache=
.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place e=
nough replicas, still in need of 1 to reach 3 (unavailableStorages=3D[DISK]=
, storagePolicy=3DBlockStoragePolicy{HOT:7, storageTypes=3D[DISK], creation=
Fallbacks=3D[], replicationFallbacks=3D[ARCHIVE]}, newBlock=3Dfalse) [</div=
></div><div><br></div><div>From one of the live 160+ DN logs, we saw=C2=A0<=
/div><div><br></div><div><div>Node /default-rack/<a href=3D"http://10.181.8=
.222:50010">10.181.8.222:50010</a> [</div><div>=C2=A0 Storage [DISK]DS-2d39=
f3c3-2e67-48ad-871b-632f66b277d7:NORMAL:<a href=3D"http://10.181.8.222:5001=
0">10.181.8.222:50010</a> is not chosen since the node is too busy (load: 2=
 &gt; 1.8370786516853932) .</div><div>]</div><div>Node /default-rack/<a hre=
f=3D"http://10.181.25.147:50010">10.181.25.147:50010</a> [</div><div>=C2=A0=
 Storage [DISK]DS-60b511b0-62aa-4c0f-92d9-6d90ff32ee49:NORMAL:<a href=3D"ht=
tp://10.181.25.147:50010">10.181.25.147:50010</a> is not chosen since the n=
ode is too busy (load: 2 &gt; 1.8370786516853932) .</div><div>]</div><div>N=
ode /default-rack/<a href=3D"http://10.181.8.152:50010">10.181.8.152:50010<=
/a> [</div><div>=C2=A0 Storage [DISK]DS-7e0bf761-86f2-4748-9eda-fbfd9c69e12=
7:NORMAL:<a href=3D"http://10.181.8.152:50010">10.181.8.152:50010</a> is no=
t chosen since the node is too busy (load: 2 &gt; 1.8370786516853932) .</di=
v><div>]</div><div>Node /default-rack/<a href=3D"http://10.181.25.67:50010"=
>10.181.25.67:50010</a> [</div><div>=C2=A0 Storage [DISK]DS-5849e4d8-4ab6-4=
392-aee2-7a354c82c19d:NORMAL:<a href=3D"http://10.181.25.67:50010">10.181.2=
5.67:50010</a> is not chosen since the node is too busy (load: 2 &gt; 1.837=
0786516853932) .</div><div>]</div></div><div><br></div><div><br></div><div>=
Few things we observed from our end</div><div>- If we restart the NN, we=
9;re able to submit jobs without any issues</div><div>- We run this Hadoop =
cluster on AWS</div><div>- DN and TT process run on a single EC2 machine wh=
ich is backed by an AutoScaling Group.</div><div>- We&#39;ve another cluste=
r which does&#39;t autoscale and doesn&#39;t exhibit the behaviour<div><div=
><br></div><div>Any pointers or ideas on how to solve this for good would b=
e really appreciated.=C2=A0</div><div><br></div>-- <br><div><div><div><br><=
/div><div>Ashwanth Kumar /=C2=A0<a href=3D"http://ashwanthkumar.in/" target=
=3D"_blank">ashwanthkumar.in</a></div><br></div></div>
</div></div></div></div>

--089e0111bea2ed9d2705248c4375--