Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of faithlessfriend@gmail.com
 designates 209.85.215.53 as permitted sender)
MIME-Version: 1.0
Date: Wed, 10 Jul 2013 15:02:00 +0300
Message-ID: 
 <CAPS2vjo2AAs7HitEzhV-U5DqD4P4MHoZx=Mu8BSOB9PtckiS4g@mail.gmail.com>
Subject: ConnectionException in container, happens only sometimes
From: Andrei <faithlessfriend@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001a11c2a4a88aa21904e1270bd9

--001a11c2a4a88aa21904e1270bd9
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I'm running CDH4.3 installation of Hadoop with the following simple setup:

master-host: runs NameNode, ResourceManager and JobHistoryServer
slave-1-host and slave-2-hosts: DataNodes and NodeManagers.

When I run simple MapReduce job (both - using streaming API or Pi example
from distribution) on client I see that some tasks fail:

13/07/10 14:40:10 INFO mapreduce.Job:  map 60% reduce 0%
13/07/10 14:40:14 INFO mapreduce.Job: Task Id :
attempt_1373454026937_0005_m_000003_0, Status : FAILED
13/07/10 14:40:14 INFO mapreduce.Job: Task Id :
attempt_1373454026937_0005_m_000005_0, Status : FAILED
...
13/07/10 14:40:23 INFO mapreduce.Job:  map 60% reduce 20%
...

Every time different set of tasks/attempts fails. In some cases number of
failed attempts becomes critical, and the whole job fails, in other cases
job is finished successfully. I can't see any dependency, but I noticed the
following.

Let's say, ApplicationMaster runs on _slave-1-host_. In this case on
_slave-2-host_ there will be corresponding syslog with the following
contents:

...
2013-07-10 11:06:10,986 INFO [main] org.apache.hadoop.ipc.Client: Retrying
connect to server: slave-2-host/127.0.0.1:11812. Already tried 0 time(s);
retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2013-07-10 11:06:11,989 INFO [main] org.apache.hadoop.ipc.Client: Retrying
connect to server: slave-2-host/127.0.0.1:11812. Already tried 1 time(s);
retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
...
2013-07-10 11:06:20,013 INFO [main] org.apache.hadoop.ipc.Client: Retrying
connect to server: slave-2-host/127.0.0.1:11812. Already tried 9 time(s);
retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2013-07-10 11:06:20,019 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.net.ConnectException: Call From slave-2-host/
127.0.0.1 to slave-2-host:11812 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
        at org.apache.hadoop.ipc.Client.call(Client.java:1229)
        at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
        at com.sun.proxy.$Proxy6.getTask(Unknown Source)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:131)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:708)
        at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)
        at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:499)
        at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:593)
        at
org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:241)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1278)
        at org.apache.hadoop.ipc.Client.call(Client.java:1196)
        ... 3 more


Notice several things:

1. This exception always happens on the different host than
ApplicationMaster runs on.
2. It always tries to connect to localhost, not other host in cluster.
3. Port number (11812 in this case) is always different.

My questions are:

1. I assume this is the task (container) that tries to establish
connection, but what it wants to connect to?
2. Why this error happens and how can I fix it?

Any suggestions are welcome.

Thanks,
Andrei

--001a11c2a4a88aa21904e1270bd9
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,=A0<div><br></div><div>I&#39;m running CDH4.3 installat=
ion of Hadoop with the following simple setup:=A0</div><div><br></div><div>=
master-host: runs NameNode, ResourceManager and JobHistoryServer</div><div>=
slave-1-host and slave-2-hosts: DataNodes and NodeManagers.=A0</div>
<div><br></div><div>When I run simple MapReduce job (both - using streaming=
 API or Pi example from distribution) on client I see that some tasks fail:=
=A0</div><div><br></div><div><div>13/07/10 14:40:10 INFO mapreduce.Job: =A0=
map 60% reduce 0%</div>
<div>13/07/10 14:40:14 INFO mapreduce.Job: Task Id : attempt_1373454026937_=
0005_m_000003_0, Status : FAILED</div><div>13/07/10 14:40:14 INFO mapreduce=
.Job: Task Id : attempt_1373454026937_0005_m_000005_0, Status : FAILED</div=
>
</div><div>...</div><div>13/07/10 14:40:23 INFO mapreduce.Job: =A0map 60% r=
educe 20%<br></div><div>...</div><div><br></div><div>Every time different s=
et of tasks/attempts fails. In some cases number of failed attempts becomes=
 critical, and the whole job fails, in other cases job is finished successf=
ully. I can&#39;t see any dependency, but I noticed the following.=A0</div>
<div><br></div><div>Let&#39;s say, ApplicationMaster runs on _slave-1-host_=
. In this case on _slave-2-host_ there will be corresponding syslog with th=
e following contents:=A0</div><div><br></div><div>...=A0</div><div><div>201=
3-07-10 11:06:10,986 INFO [main] org.apache.hadoop.ipc.Client: Retrying con=
nect to server: slave-2-host/<a href=3D"http://127.0.0.1:11812">127.0.0.1:1=
1812</a>. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWit=
hFixedSleep(maxRetries=3D10, sleepTime=3D1 SECONDS)</div>
<div>2013-07-10 11:06:11,989 INFO [main] org.apache.hadoop.ipc.Client: Retr=
ying connect to server: slave-2-host/<a href=3D"http://127.0.0.1:11812">127=
.0.0.1:11812</a>. Already tried 1 time(s); retry policy is RetryUpToMaximum=
CountWithFixedSleep(maxRetries=3D10, sleepTime=3D1 SECONDS)</div>
</div><div>...</div><div><div>2013-07-10 11:06:20,013 INFO [main] org.apach=
e.hadoop.ipc.Client: Retrying connect to server: slave-2-host/<a href=3D"ht=
tp://127.0.0.1:11812">127.0.0.1:11812</a>. Already tried 9 time(s); retry p=
olicy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3D10, sleepTime=3D1=
 SECONDS)</div>
<div>2013-07-10 11:06:20,019 WARN [main] org.apache.hadoop.mapred.YarnChild=
: Exception running child : java.net.ConnectException: Call From slave-2-ho=
st/<a href=3D"http://127.0.0.1">127.0.0.1</a> to slave-2-host:11812 failed =
on connection exception: java.net.ConnectException: Connection refused; For=
 more details see: =A0<a href=3D"http://wiki.apache.org/hadoop/ConnectionRe=
fused">http://wiki.apache.org/hadoop/ConnectionRefused</a></div>
<div>=A0 =A0 =A0 =A0 at sun.reflect.NativeConstructorAccessorImpl.newInstan=
ce0(Native Method)</div><div>=A0 =A0 =A0 =A0 at sun.reflect.NativeConstruct=
orAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)</div><div=
>=A0 =A0 =A0 =A0 at sun.reflect.DelegatingConstructorAccessorImpl.newInstan=
ce(DelegatingConstructorAccessorImpl.java:45)</div>
<div>=A0 =A0 =A0 =A0 at java.lang.reflect.Constructor.newInstance(Construct=
or.java:526)</div><div>=A0 =A0 =A0 =A0 at org.apache.hadoop.net.NetUtils.wr=
apWithMessage(NetUtils.java:782)</div><div>=A0 =A0 =A0 =A0 at org.apache.ha=
doop.net.NetUtils.wrapException(NetUtils.java:729)</div>
<div>=A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.Client.call(Client.java:1229)=
</div><div>=A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.WritableRpcEngine$Invok=
er.invoke(WritableRpcEngine.java:225)</div><div>=A0 =A0 =A0 =A0 at com.sun.=
proxy.$Proxy6.getTask(Unknown Source)</div>
<div>=A0 =A0 =A0 =A0 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.j=
ava:131)</div><div>Caused by: java.net.ConnectException: Connection refused=
</div><div>=A0 =A0 =A0 =A0 at sun.nio.ch.SocketChannelImpl.checkConnect(Nat=
ive Method)</div>
<div>=A0 =A0 =A0 =A0 at sun.nio.ch.SocketChannelImpl.finishConnect(SocketCh=
annelImpl.java:708)</div><div>=A0 =A0 =A0 =A0 at org.apache.hadoop.net.Sock=
etIOWithTimeout.connect(SocketIOWithTimeout.java:207)</div><div>=A0 =A0 =A0=
 =A0 at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)</div>
<div>=A0 =A0 =A0 =A0 at org.apache.hadoop.net.NetUtils.connect(NetUtils.jav=
a:492)</div><div>=A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.Client$Connection=
.setupConnection(Client.java:499)</div><div>=A0 =A0 =A0 =A0 at org.apache.h=
adoop.ipc.Client$Connection.setupIOstreams(Client.java:593)</div>
<div>=A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.Client$Connection.access$2000=
(Client.java:241)</div><div>=A0 =A0 =A0 =A0 at org.apache.hadoop.ipc.Client=
.getConnection(Client.java:1278)</div><div>=A0 =A0 =A0 =A0 at org.apache.ha=
doop.ipc.Client.call(Client.java:1196)</div>
<div>=A0 =A0 =A0 =A0 ... 3 more</div></div><div><br></div><div><br></div><d=
iv>Notice several things:=A0</div><div><br></div><div>1. This exception alw=
ays happens on the different host than ApplicationMaster runs on.=A0</div><=
div>2. It always tries to connect to localhost, not other host in cluster.=
=A0</div>
<div>3. Port number (11812 in this case) is always different.=A0</div><div>=
<br></div><div>My questions are:=A0</div><div><br></div><div>1. I assume th=
is is the task (container) that tries to establish connection, but what it =
wants to connect to?=A0</div>
<div>2. Why this error happens and how can I fix it?=A0</div><div><br></div=
><div>Any suggestions are welcome.</div><div><br></div><div>Thanks,=A0</div=
><div>Andrei</div></div>

--001a11c2a4a88aa21904e1270bd9--