Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of psybers@gmail.com designates
 209.85.215.51 as permitted sender)
MIME-Version: 1.0
Reply-To: rdyer@iastate.edu
Date: Mon, 21 Oct 2013 23:55:45 -0500
Message-ID: 
 <CAMiz3FMQv_9SVPKA1QKuJEBOvpNK1qmVXmnX_oOGyNSD=USefw@mail.gmail.com>
Subject: Hadoop 2.2.0 MR tasks failing
From: Robert Dyer <psybers@gmail.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=089e0115fe1eac602404e94d365c

--089e0115fe1eac602404e94d365c
Content-Type: text/plain; charset=ISO-8859-1

I recently setup a 2.2.0 test cluster.  For some reason, all of my MR jobs
are failing.  The maps and reduces all run to completion, without any
errors.  Yet the app is marked failed and there is no final output.  Any
ideas?

Application Type: MAPREDUCE
State: FINISHED
FinalStatus: FAILED
Diagnostics: We crashed durring a commit

I notice in the logs this (but not sure what to make of it):

2013-10-21 23:42:41,379 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
Memory usage of ProcessTree 789 for container-id
container_1382415258498_0002_01_000001: 250.4 MB of 2 GB physical
memory used; 2.0 GB of 6 GB virtual memory used
2013-10-21 23:42:41,743 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Exit code from container container_1382415258498_0002_01_000001 is :
255
2013-10-21 23:42:41,744 WARN
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Exception from container-launch with container ID:
container_1382415258498_0002_01_000001 and exit code: 255
org.apache.hadoop.util.Shell$ExitCodeException:

2013-10-21 23:42:41,746 INFO
org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:
2013-10-21 23:42:41,747 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Container exited with a non-zero exit code 255
2013-10-21 23:42:41,747 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container:
Container container_1382415258498_0002_01_000001 transitioned from
RUNNING to EXITED_WITH_FAILURE
2013-10-21 23:42:41,747 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch:
Cleaning up container container_1382415258498_0002_01_000001
2013-10-21 23:42:41,764 INFO
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
Deleting absolute path :
/hadoop/hadoop-2.2.0/cluster-data/usercache/hadoop/appcache/application_1382415258498_0002/container_1382415258498_0002_01_000001
2013-10-21 23:42:41,765 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger:
USER=hadoop	OPERATION=Container Finished -
Failed	TARGET=ContainerImpl	RESULT=FAILURE	DESCRIPTION=Container
failed with state:
EXITED_WITH_FAILURE	APPID=application_1382415258498_0002	CONTAINERID=container_1382415258498_0002_01_000001

--089e0115fe1eac602404e94d365c
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>I recently setup a 2.2.0 test cluster. =A0For some re=
ason, all of my MR jobs are failing. =A0The maps and reduces all run to com=
pletion, without any errors. =A0Yet the app is marked failed and there is n=
o final output. =A0Any ideas?</div>
<div><br></div>Application Type: MAPREDUCE<br>State: FINISHED<br>FinalStatu=
s: FAILED<br>Diagnostics:=A0We crashed durring a commit<br><div><br></div><=
div>I notice in the logs this (but not sure what to make of it):</div><div>
<pre style=3D"color:rgb(0,0,0);word-wrap:break-word;white-space:pre-wrap">2=
013-10-21 23:42:41,379 INFO org.apache.hadoop.yarn.server.nodemanager.conta=
inermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 789 =
for container-id container_1382415258498_0002_01_000001: 250.4 MB of 2 GB p=
hysical memory used; 2.0 GB of 6 GB virtual memory used
2013-10-21 23:42:41,743 WARN org.apache.hadoop.yarn.server.nodemanager.Defa=
ultContainerExecutor: Exit code from container container_1382415258498_0002=
_01_000001 is : 255
2013-10-21 23:42:41,744 WARN org.apache.hadoop.yarn.server.nodemanager.Defa=
ultContainerExecutor: Exception from container-launch with container ID: co=
ntainer_1382415258498_0002_01_000001 and exit code: 255
org.apache.hadoop.util.Shell$ExitCodeException: </pre><pre style=3D"color:r=
gb(0,0,0);word-wrap:break-word;white-space:pre-wrap"><pre style=3D"word-wra=
p:break-word;white-space:pre-wrap">2013-10-21 23:42:41,746 INFO org.apache.=
hadoop.yarn.server.nodemanager.ContainerExecutor:=20
2013-10-21 23:42:41,747 WARN org.apache.hadoop.yarn.server.nodemanager.cont=
ainermanager.launcher.ContainerLaunch: Container exited with a non-zero exi=
t code 255
2013-10-21 23:42:41,747 INFO org.apache.hadoop.yarn.server.nodemanager.cont=
ainermanager.container.Container: Container container_1382415258498_0002_01=
_000001 transitioned from RUNNING to EXITED_WITH_FAILURE
2013-10-21 23:42:41,747 INFO org.apache.hadoop.yarn.server.nodemanager.cont=
ainermanager.launcher.ContainerLaunch: Cleaning up container container_1382=
415258498_0002_01_000001
2013-10-21 23:42:41,764 INFO org.apache.hadoop.yarn.server.nodemanager.Defa=
ultContainerExecutor: Deleting absolute path : /hadoop/hadoop-2.2.0/cluster=
-data/usercache/hadoop/appcache/application_1382415258498_0002/container_13=
82415258498_0002_01_000001
2013-10-21 23:42:41,765 WARN org.apache.hadoop.yarn.server.nodemanager.NMAu=
ditLogger: USER=3Dhadoop	OPERATION=3DContainer Finished - Failed	TARGET=3DC=
ontainerImpl	RESULT=3DFAILURE	DESCRIPTION=3DContainer failed with state: EX=
ITED_WITH_FAILURE	APPID=3Dapplication_1382415258498_0002	CONTAINERID=3Dcont=
ainer_1382415258498_0002_01_000001</pre>
</pre></div></div>

--089e0115fe1eac602404e94d365c--