hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Out of memory after Map tasks
Date Thu, 25 May 2006 19:09:23 GMT
Vijay Murthi wrote:
>>Are you running the current trunk?  My guess is that you are.  If so,
>>then this error is "normal", things should keep running.
> 
> I am using hadoop-0.2.0. I believe this is the current trunk.

No, that's a release.  The trunk is what's currently in Subversion.

> I used to
> think child task exit with "Out of memory" is normal since the job can
> be re-executed on another machine that finish whereas Tasktracker which
> manages should not. After this message I see only one Tasktracker
> running on each node with "99%" on CPU all the time not any reduce task.

It sounds like these "Out of memory" errors are fatal.

> On "mapred" local directory I see it writing to directory of name
> "*_r_*". Since every output map task produce is on local disk can't it
> just read those reduce files Map task create?

The local map output files are mostly needed by reduces running on other 
nodes and must first be transferred.

> I am running on 64 bit kernel with JVM set to 32-bit. The JAVA heap size
> set to a maximum of 1 GB for both Tasktracker and child process. I
> believe Tasktracker and each child process runs on its own JVM of 1 GB
> (correct, me if I am wrong).  Does each child process should have less
> memory than Tasktracker or total of memory of child process it manage
> should be less than Tasktracker memory heap since Tasktracker creates
> the children? In my case, I am setting 500 MB of sort memory for each
> child reduce process. So 3 reduce task * 500 MB can be more than 1 GB
> and causes "Out of memory"?

Why are you using 500MB of sort memory with a 1GB heap if it keeps 
causing problems?  I would suggest either decreasing the sort memory or 
increasing the heap size.  Better yet, start with the defaults and 
change one parameter at a time.

>>4MB buffers for file streams seems large to me.
> 
> I keep 4 MB buffer because each Map task reading around 2 GB Gzip text
> file. I thought this will make the reading process efficient and 4 MB *
> 3 map task per node is like 12 MB. Not sure, why this is lot.

Again, changing one setting at a time will allow you to better figure 
out what improves things and what causes problems.  This parameter is 
used for lots of files, more than just your input data, so increasing it 
to 4MB causes lots of 4MB buffers to be created.  I have a hard time 
seeing a justification ever for buffers larger than 1MB, as even 100k 
should usually cause transfer to dominate seek, but, since map and 
reduce both operate sequentially, even 100k should not be required for 
good performance.

>>So you could even use a sort factor of 500.  That would
>>make sorts a lot faster.
> 
> Ok I will try that. I have around 120 reduce files in total each around
> 1 GB for 6 reduce process. 

Please first try things with the defaults.  Then try increasing the sort 
factor to find if that improves things for you.

>>Also why are you setting the task timeout so high?  Do you have
> 
> mappers
> 
>>or reducers that take a long time per entry and are not calling
>>Reporter.setStatus() regularly?  That can cause tasks to time out.
> 
> Yes. Map task sometime take a long time and got killed. I have a
> reporter that set status when record reader is created. Still things get
> printed on the web page only after the task exit with Succeed or Failure
> status. 

If processing a single record could take longer than the task timeout 
(10 minutes) then you should call setStatus() during the processing of 
the record to avoid timeouts.  That's a better way to fix this than to 
increase the task timeout.  Note that setStatus() is efficient: don't 
worry about calling it too often.

Doug

Mime
View raw message