Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: 209.85.220.176 is neither permitted
 nor denied by domain of niranjan@cs.washington.edu)
From: Niranjan Balasubramanian <niranjan@cs.washington.edu>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_5935EF32-8418-4F20-BE3A-3099EC50C828"
Subject: OOM Error Map output copy.
Date: Wed, 7 Dec 2011 10:51:12 -0800
Message-Id: <349A4555-E892-4A67-946D-D07C6732E960@cs.washington.edu>
To: common-user@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1244.3)

--Apple-Mail=_5935EF32-8418-4F20-BE3A-3099EC50C828
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

All=20

I am encountering the following out-of-memory error during the reduce =
phase of a large job.

Map output copy failure : java.lang.OutOfMemoryError: Java heap space
	at =
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleIn=
Memory(ReduceTask.java:1669)
	at =
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOut=
put(ReduceTask.java:1529)
	at =
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutpu=
t(ReduceTask.java:1378)
	at =
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(Reduc=
eTask.java:1310)
I tried increasing the memory available using mapped.child.java.opts but =
that only helps a little. The reduce task eventually fails again. Here =
are some relevant job configuration details:

1. The input to the mappers is about 2.5 TB (LZO compressed). The =
mappers filter out a small percentage of the input ( less than 1%).

2. I am currently using 12 reducers and I can't increase this count by =
much to ensure availability of reduce slots for other users.=20

3. mapred.child.java.opts --> -Xms512M -Xmx1536M -XX:+UseSerialGC

4. mapred.job.shuffle.input.buffer.percent	--> 0.70

5. mapred.job.shuffle.merge.percent	--> 0.66

6. mapred.inmem.merge.threshold	--> 1000

7. I have nearly 5000 mappers which are supposed to produce LZO =
compressed outputs. The logs seem to indicate that the map outputs range =
between 0.3G to 0.8GB.=20

Does anything here seem amiss? I'd appreciate any input of what settings =
to try. I can try different reduced values for the input buffer percent =
and the merge percent.  Given that the job runs for about 7-8 hours =
before crashing, I would like to make some informed choices if possible.

Thanks.=20
~ Niranjan.


--Apple-Mail=_5935EF32-8418-4F20-BE3A-3099EC50C828--