hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris MacKenzie <stu...@chrismackenziephotography.co.uk>
Subject Re: Configuration set up questions - Container killed on request. Exit code is 143
Date Fri, 18 Jul 2014 09:52:56 GMT
Hi Guys,

Thanks very much for getting back to me.
 

Thanks Chris - the idea of slitting the data is a great suggestion.
Yes Wangda, I was restarting after changing the configs

I’ve been checking the relationship between what I thought was in my
config files and what hadoop thought were in them.

With:

// Print out Config file settings for testing.
		for (Entry<String, String> entry: conf){
	System.out.printf("%s=%s\n", entry.getKey(), entry.getValue());
		}



There were anomalies ;0(

Now that my hadoop reflects the values that are in my config files - I
just get the message “Killed” without any explanation.


Unfortunately, where I was applying changes incrementally and testing I’ve
applied all the changes all at once.

I’m now backing out the changes I made slowly to see where it starts to
reflect what I expect.

Regards,

Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>






From:  Chris Mawata <chris.mawata@gmail.com>
Reply-To:  <user@hadoop.apache.org>
Date:  Thursday, 17 July 2014 16:15
To:  <user@hadoop.apache.org>
Subject:  Re: Configuration set up questions - Container killed on
request. Exit code is 143


Another thing to try is smaller input splits if your data can be broken up
into smaller files that can be independently processed. That way s
you get more but smaller map tasks. You could also use more  but smaller
reducers. The many files will tax your NameNode more but you might get to
use all you cores.
On Jul 17, 2014 9:07 AM, "Chris MacKenzie"
<studio@chrismackenziephotography.co.uk> wrote:

Hi Chris,

Thanks for getting back to me. I will set that value to 10

I have just tried this.
https://support.gopivotal.com/hc/en-us/articles/201462036-Mapreduce-YARN-Me
mory-Parameters

Setting both to mapreduce.map.memory.mb mapreduce.reduce.memory.mb. Though
after setting it I didn’t get the expected change.

As the output was still 2.1 GB of 2.1 GB virtual memory used. Killing
container


Regards,

Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk>
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>






From:  Chris Mawata <chris.mawata@gmail.com>
Reply-To:  <user@hadoop.apache.org>
Date:  Thursday, 17 July 2014 13:36
To:  Chris MacKenzie <studio@chrismackenziephotography.co.uk>
Cc:  <user@hadoop.apache.org>
Subject:  Re: Configuration set up questions - Container killed on
request. Exit code is 143


Hi Chris MacKenzie,      I have a feeling (I am not familiar with the kind
of work you are doing) that your application is memory intensive.  8 cores
per node and only 12GB is tight. Try bumping up the
yarn.nodemanager.vmem-pmem-ratio
Chris Mawata




On Wed, Jul 16, 2014 at 11:37 PM, Chris MacKenzie
<studio@chrismackenziephotography.co.uk> wrote:

Hi,

Thanks Chris Mawata
I’m working through this myself, but wondered if anyone could point me in
the right direction.

I have attached my configs.


I’m using hadoop 2.41

My system is:
32 Clusters
8 processors per machine
12 gb ram
Available disk space per node 890 gb

This is my current error:

mapreduce.Job (Job.java:printTaskEvents(1441)) - Task Id :
attempt_1405538067846_0006_r_000000_1, Status : FAILED
Container [pid=25848,containerID=container_1405538067846_0006_01_000004]
is running beyond virtual memory limits. Current usage: 439.0 MB of 1 GB
physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing
container.
Dump of the process-tree for container_1405538067846_0006_01_000004 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 25853 25848 25848 25848 (java) 2262 193 2268090368 112050
/usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN -Xmx768m
-Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap
plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog
s/application_1405538067846_0006/container_1405538067846_0006_01_000004
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056
attempt_1405538067846_0006_r_000000_1 4
        |- 25848 25423 25848 25848 (bash) 0 0 108613632 333 /bin/bash -c
/usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true
-Dhadoop.metrics.log.level=WARN  -Xmx768m
-Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap
plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog
s/application_1405538067846_0006/container_1405538067846_0006_01_000004
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056
attempt_1405538067846_0006_r_000000_1 4
1>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846
_0006/container_1405538067846_0006_01_000004/stdout
2>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846
_0006/container_1405538067846_0006_01_000004/stderr

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143






Regards,

Chris MacKenzie
telephone: 0131 332 6967
email: studio@chrismackenziephotography.co.uk
corporate: www.chrismackenziephotography.co.uk
<http://www.chrismackenziephotography.co.uk>
<http://www.chrismackenziephotography.co.uk>
<http://www.chrismackenziephotography.co.uk/>
<http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
<http://www.linkedin.com/in/chrismackenziephotography/>






From:  Chris Mawata <chris.mawata@gmail.com>
Reply-To:  <user@hadoop.apache.org>
Date:  Thursday, 17 July 2014 02:10
To:  <user@hadoop.apache.org>
Subject:  Re: Can someone shed some light on this ? - java.io.IOException:
Spill failed


I would post the configuration files -- easier for someone to spot
something wrong than to imagine what configuration would get you to that
stacktrace. The part
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
attempt_1405523201400_0006_m_000000_0_spill_8.out

would suggest you might not have hadoop.tmp.dir set (?)



On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie
<studio@chrismackenziephotography.co.uk> wrote:

Hi,

Is this a coding or a setup issue ?

I¹m using Hadoop 2.41
My program is doing a concordance on 500,000 sequences of 400 chars.
My cluster set is 32 data nodes and two masters.

The exact error is:
Error: java.io.IOException: Spill failed
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas
k.java:1535)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062)
        at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
        at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput
OutputContextImpl.java:89)
        at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp
er.java:112)
        at
par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
96)
        at
par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
1)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
ava:1556)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
not find any valid local directory for
attempt_1405523201400_0006_m_000000_0_spill_8.out
        at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW
rite(LocalDirAllocator.java:402)
        at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
or.java:150)
        at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
or.java:131)
        at
org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil
es.java:159)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
1566)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85
3)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja
va:1505)

Regards,

Chris
















Mime
View raw message