hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Mawata <chris.maw...@gmail.com>
Subject Re: Configuration set up questions - Container killed on request. Exit code is 143
Date Thu, 17 Jul 2014 15:15:51 GMT
Another thing to try is smaller input splits if your data can be broken up
into smaller files that can be independently processed. That way s
you get more but smaller map tasks. You could also use more  but smaller
reducers. The many files will tax your NameNode more but you might get to
use all you cores.
On Jul 17, 2014 9:07 AM, "Chris MacKenzie" <
studio@chrismackenziephotography.co.uk> wrote:

> Hi Chris,
>
> Thanks for getting back to me. I will set that value to 10
>
> I have just tried this.
> https://support.gopivotal.com/hc/en-us/articles/201462036-Mapreduce-YARN-Me
> mory-Parameters
>
> Setting both to mapreduce.map.memory.mb mapreduce.reduce.memory.mb. Though
> after setting it I didn’t get the expected change.
>
> As the output was still 2.1 GB of 2.1 GB virtual memory used. Killing
> container
>
>
> Regards,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>
>
>
>
> From:  Chris Mawata <chris.mawata@gmail.com>
> Reply-To:  <user@hadoop.apache.org>
> Date:  Thursday, 17 July 2014 13:36
> To:  Chris MacKenzie <studio@chrismackenziephotography.co.uk>
> Cc:  <user@hadoop.apache.org>
> Subject:  Re: Configuration set up questions - Container killed on
> request. Exit code is 143
>
>
> Hi Chris MacKenzie,      I have a feeling (I am not familiar with the kind
> of work you are doing) that your application is memory intensive.  8 cores
> per node and only 12GB is tight. Try bumping up the
> yarn.nodemanager.vmem-pmem-ratio
> Chris Mawata
>
>
>
>
> On Wed, Jul 16, 2014 at 11:37 PM, Chris MacKenzie
> <studio@chrismackenziephotography.co.uk> wrote:
>
> Hi,
>
> Thanks Chris Mawata
> I’m working through this myself, but wondered if anyone could point me in
> the right direction.
>
> I have attached my configs.
>
>
> I’m using hadoop 2.41
>
> My system is:
> 32 Clusters
> 8 processors per machine
> 12 gb ram
> Available disk space per node 890 gb
>
> This is my current error:
>
> mapreduce.Job (Job.java:printTaskEvents(1441)) - Task Id :
> attempt_1405538067846_0006_r_000000_1, Status : FAILED
> Container [pid=25848,containerID=container_1405538067846_0006_01_000004]
> is running beyond virtual memory limits. Current usage: 439.0 MB of 1 GB
> physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing
> container.
> Dump of the process-tree for container_1405538067846_0006_01_000004 :
>         |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>         |- 25853 25848 25848 25848 (java) 2262 193 2268090368 112050
> /usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN -Xmx768m
> -Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap
> plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog
> s/application_1405538067846_0006/container_1405538067846_0006_01_000004
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056
> attempt_1405538067846_0006_r_000000_1 4
>         |- 25848 25423 25848 25848 (bash) 0 0 108613632 333 /bin/bash -c
> /usr/java/latest//bin/java -Djava.net.preferIPv4Stack=true
> -Dhadoop.metrics.log.level=WARN  -Xmx768m
> -Djava.io.tmpdir=/tmp/hadoop-cm469/nm-local-dir/usercache/cm469/appcache/ap
> plication_1405538067846_0006/container_1405538067846_0006_01_000004/tmp
> -Dlog4j.configuration=container-log4j.properties
> -Dyarn.app.container.log.dir=/scratch/extra/cm469/hadoop-2.4.1/logs/userlog
> s/application_1405538067846_0006/container_1405538067846_0006_01_000004
> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
> org.apache.hadoop.mapred.YarnChild 137.195.143.103 59056
> attempt_1405538067846_0006_r_000000_1 4
> 1>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846
> _0006/container_1405538067846_0006_01_000004/stdout
> 2>/scratch/extra/cm469/hadoop-2.4.1/logs/userlogs/application_1405538067846
> _0006/container_1405538067846_0006_01_000004/stderr
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
>
>
>
>
>
> Regards,
>
> Chris MacKenzie
> telephone: 0131 332 6967
> email: studio@chrismackenziephotography.co.uk
> corporate: www.chrismackenziephotography.co.uk
> <http://www.chrismackenziephotography.co.uk>
> <http://www.chrismackenziephotography.co.uk/>
> <http://plus.google.com/+ChrismackenziephotographyCoUk/posts>
> <http://www.linkedin.com/in/chrismackenziephotography/>
>
>
>
>
>
>
> From:  Chris Mawata <chris.mawata@gmail.com>
> Reply-To:  <user@hadoop.apache.org>
> Date:  Thursday, 17 July 2014 02:10
> To:  <user@hadoop.apache.org>
> Subject:  Re: Can someone shed some light on this ? - java.io.IOException:
> Spill failed
>
>
> I would post the configuration files -- easier for someone to spot
> something wrong than to imagine what configuration would get you to that
> stacktrace. The part
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
> not find any valid local directory for
> attempt_1405523201400_0006_m_000000_0_spill_8.out
>
> would suggest you might not have hadoop.tmp.dir set (?)
>
>
>
> On Wed, Jul 16, 2014 at 1:02 PM, Chris MacKenzie
> <studio@chrismackenziephotography.co.uk> wrote:
>
> Hi,
>
> Is this a coding or a setup issue ?
>
> I¹m using Hadoop 2.41
> My program is doing a concordance on 500,000 sequences of 400 chars.
> My cluster set is 32 data nodes and two masters.
>
> The exact error is:
> Error: java.io.IOException: Spill failed
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.checkSpillException(MapTas
> k.java:1535)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1062)
>         at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
>         at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInput
> OutputContextImpl.java:89)
>         at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapp
> er.java:112)
>         at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 96)
>         at
> par.gene.align.v3.concordance.ConcordanceMapper.map(ConcordanceMapper.java:
> 1)
>         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.j
> ava:1556)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could
> not find any valid local directory for
> attempt_1405523201400_0006_m_000000_0_spill_8.out
>         at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForW
> rite(LocalDirAllocator.java:402)
>         at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:150)
>         at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocat
> or.java:131)
>         at
> org.apache.hadoop.mapred.YarnOutputFiles.getSpillFileForWrite(YarnOutputFil
> es.java:159)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:
> 1566)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$900(MapTask.java:85
> 3)
>         at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.ja
> va:1505)
>
> Regards,
>
> Chris
>
>
>
>
>
>
>
>
>
>
>
>
>
>

Mime
View raw message