hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris MacKenzie <stu...@chrismackenziephotography.co.uk>
Subject Configuration set up questions - Container killed on request. Exit code is 143
Date Thu, 17 Jul 2014 03:37:47 GMT
Hi,

Thanks Chris Mawata
I’m working through this myself, but wondered if anyone could point me in
the right direction.

I have attached my configs.


I’m using hadoop 2.41

My system is:
32 Clusters
8 processors per machine
12 gb ram
Available disk space per node 890 gb

This is my current error:

mapreduce.Job (Job.java:printTaskEvents(1441)) - Task Id :
attempt_1405538067846_0006_r_000000_1, Status : FAILED
Container [pid=25848,containerID=container_1405538067846_0006_01_000004]
is running beyond virtual memory limits. Current usage: 439.0 MB of 1 GB
physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing
container.
Dump of the process-tree for container_1405538067846_0006_01_000004 :
	|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
	|- 25853 25848 25848 25848 (java) 2262 193 2268090368 112050
/usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN -Xmx768m
-Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap
plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog
s/application_1405538067846_0006/container_1405538067846_0006_01_000004
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056
attempt_1405538067846_0006_r_000000_1 4
	|- 25848 25423 25848 25848 (bash) 0 0 108613632 333 /bin/bash -c
/usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN  -Xmx768m
-Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap
plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog
s/application_1405538067846_0006/container_1405538067846_0006_01_000004
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056
attempt_1405538067846_0006_r_000000_1 4
1>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846
_0006/container_1405538067846_0006_01_000004/stdout
2>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846
_0006/container_1405538067846_0006_01_000004/stderr

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143






Regards,

Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>






From:  Chris Mawata <chris.mawata@gmail.com>
Reply-To:  <user@hadoop.apache.org>
Date:  Thursday, 17 July 2014 02:10
To:  <user@hadoop.apache.org>
Subject:  Re: Can someone shed some light on this ? - java.io.IOException:
Spill failed


I would post the configuration files -- easier for someone to spot
something wrong than to imagine what configuration would get you to that
stacktrace. The part
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
attempt_1405523201400_0006_m_000000_0_spill_8.out

would suggest you might not have hadoop.tmp.dir set (?)



On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie
<studio@chrismackenziephotography.co.uk> wrote:

Hi,

Is this a coding or a setup issue ?

I¹m using Hadoop 2.41
My program is doing a concordance on 500,000 sequences of 400 chars.
My cluster set is 32 data nodes and two masters.

The exact error is:
Error: java.io.IOException: Spill failed
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas
k.java:1535)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062)
        at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
        at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput
OutputContextImpl.java:89)
        at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp
er.java:112)
        at
par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
96)
        at
par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
1)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
ava:1556)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
attempt_1405523201400_0006_m_000000_0_spill_8.out
        at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW
rite(LocalDirAllocator.java:402)
        at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
or.java:150)
        at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
or.java:131)
        at
org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil
es.java:159)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
1566)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85
3)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja
va:1505)

Regards,

Chris








Mime
View raw message