hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom White <...@cloudera.com>
Subject Re: move tasks to another machine on the fly
Date Wed, 06 May 2009 08:59:46 GMT
Hi David,

The MapReduce framework will attempt to rerun failed tasks
automatically. However, if a task is running out of memory on one
machine, it's likely to run out of memory on another, isn't it? Have a
look at the mapred.child.java.opts configuration property for the
amount of memory that each task VM is given (200MB by default). You
can also control the memory that each daemon gets using the
HADOOP_HEAPSIZE variable in hadoop-env.sh. Or you can specify it on a
per-daemon basis using the HADOOP_<DAEMON_NAME>_OPTS variables in the
same file.

Tom

On Wed, May 6, 2009 at 1:28 AM, David Batista <dsbatista@gmail.com> wrote:
> I get this error when running Reduce tasks on a machine:
>
> java.lang.OutOfMemoryError: unable to create new native thread
>        at java.lang.Thread.start0(Native Method)
>        at java.lang.Thread.start(Thread.java:597)
>        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.(DFSClient.java:2591)
>        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:454)
>        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:190)
>        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:487)
>        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:387)
>        at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:117)
>        at org.apache.hadoop.mapred.lib.MultipleTextOutputFormat.getBaseRecordWriter(MultipleTextOutputFormat.java:44)
>        at org.apache.hadoop.mapred.lib.MultipleOutputFormat$1.write(MultipleOutputFormat.java:99)
>        at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:410)
>
> is it possible to move a reduce task to other machine in the cluster on the fly?
>
> --
> ./david
>

Mime
View raw message