Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <539205.6484.qm@web65507.mail.ac4.yahoo.com>
Date: Sat, 18 Apr 2009 06:39:41 -0700 (PDT)
From: Andrew Purtell <apurtell@apache.org>
Reply-To: apurtell@apache.org
Subject: Re: Ec2 instability
To: hbase-user@hadoop.apache.org, core-user@hadoop.apache.org
In-Reply-To: <384813770904170944l1b3843c8v6b32970b8391831b@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii


Hi,

This is an OS level exception. Your node is out of memory
even to fork a process. 

How many instances do you currently have allocated? Have
you increased the number of instances over time to try and
spread the load of your application around? How many
concurrent mapper and/or reducer processes do you execute
on a node? Can you characterize the memory usage of your
mappers and reducers? Are you running other processes
external to hadoop/hbase which consume a lot of memory? Are
you running Ganglia or similar to track and characterize
resource usage over time? 

You may find you are trying to solve a 100 node problem
with 10.

   - Andy

> From: Rakhi Khatwani
> Subject: Re: Ec2 instability
> To: hbase-user@hadoop.apache.org, core-user@hadoop.apache.org
> Date: Friday, April 17, 2009, 9:44 AM
> Hi,
>  this is the exception i have been getting @ the mapreduce
> 
> java.io.IOException: Cannot run program "bash":
> java.io.IOException:
> error=12, Cannot allocate memory
> 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
> 	at org.apache.hadoop.util.Shell.runCommand(Shell.java:149)
> 	at org.apache.hadoop.util.Shell.run(Shell.java:134)
> 	at org.apache.hadoop.fs.DF.getAvailable(DF.java:73)
> 	at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:321)
> 	at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
> 	at
> org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:61)
> 	at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1199)
> 	at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:857)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
> 	at org.apache.hadoop.mapred.Child.main(Child.java:155)
> Caused by: java.io.IOException: java.io.IOException:
> error=12, Cannot
> allocate memory
> 	at java.lang.UNIXProcess.(UNIXProcess.java:148)
> 	at java.lang.ProcessImpl.start(ProcessImpl.java:65)
> 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
> 	... 10 more