hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Didonna <m.didonn...@gmail.com>
Subject Re: [EC2] Could not reserve enough space for object heap
Date Mon, 12 Dec 2011 08:40:27 GMT
Hi, thanks for your answer.
I am indeed using a 64bit jvm on all nodes. In fact I get the
following answer when executing java -version

java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

And I do have a looot of free RAM:

                   total       used       free     shared    buffers     cached
Mem:          7700        930       6769          0         14        462
-/+ buffers/cache:        453       7246
Swap:            0          0          0


And still I cannot make use of such amount of ram. Again I tried to
set mapred.child.java.opts to -Xmx1g but the map task got only the
default  amount of heap space (I detect this using the ps command).
Only If i use mapred.{map|reduce}.child.java.opts=-Xmx1g this setting
affects the job execution and I end up with the error we're trying to
debug.
The other non-default setting can be seen from the attached whirr
configuration file (previous message).

Thanks again,
Marco

On 11 December 2011 21:37, Harsh J <harsh@cloudera.com> wrote:
> Marco,
>
> Are you using a 64-bit JVM on your nodes or a 32-bit one?
>
> Sun JRE should say something like (for hadoop -version):
> "Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)"
>
> If you are, could you post what 'free' says on your slave nodes?
>
> On Sun, Dec 11, 2011 at 11:29 PM, Marco Didonna <m.didonna86@gmail.com> wrote:
>> Hello everyone,
>> I'm running a small toy cluster (3 nodes) on EC2 configured as follows:
>>
>> * one node as JT+NN
>> * two nodes as DN+TT
>>
>> I use whirr to build such cluster on demand (config file here
>> http://pastebin.com/JXHYvMNb). Since my jobs are memory intensive I'd
>> like to exploit the 8GB of ram the m1.large instance offers. Thus I
>> added mapred.map.child.java.opts=-Xmx1g and
>> mapred.reduce.child.java.opts=-Xmx1g (adding just
>> hadoop-mapreduce.mapred.child.java.opts=-Xmx1g produced no effect
>> since the map tasks were allocated the default 200MB). The problem is
>> that with these settings I cannot have any job running because I
>> always get
>>
>> 1/12/11 18:14:24 INFO mapred.JobClient: Running job: job_201112111644_0002
>> 11/12/11 18:14:25 INFO mapred.JobClient:  map 0% reduce 0%
>> 11/12/11 18:14:28 INFO mapred.JobClient: Task Id :
>> attempt_201112111644_0002_m_000004_0, Status : FAILED
>> java.lang.Throwable: Child Error
>>        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
>>        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>>
>> 11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
>> outputip-10-87-1-170.ec2.internal
>> 11/12/11 18:14:28 WARN mapred.JobClient: Error reading task
>> outputip-10-87-1-170.ec2.internal
>> 11/12/11 18:14:30 INFO mapred.JobClient: Task Id :
>> attempt_201112111644_0002_r_000001_0, Status : FAILED
>> java.lang.Throwable: Child Error
>>        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:242)
>> Caused by: java.io.IOException: Task process exit with nonzero status of 1.
>>        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
>>
>> And the only log I can get from the job_201112111644_0002 is the
>> stdout and the stderr whose combined output is
>>
>> Could not create the Java virtual machine.
>> Error occurred during initialization of VM
>> Could not reserve enough space for object heap
>>
>> I really cannot understand why the jvm cannot allocate enough space:
>> there's plenty of ram. I also tried to reduce the number of map slots
>> to two: nothing changed. I'm out of ideas. I hope you can shed some
>> light :)
>>
>> FYI I use cloudera distribution for hadoop, latest stable release available.
>>
>> Thanks for your attention.
>>
>> MD
>
>
>
> --
> Harsh J

Mime
View raw message