Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0711910620 for ; Thu, 3 Apr 2014 09:50:03 +0000 (UTC) Received: (qmail 47317 invoked by uid 500); 3 Apr 2014 09:49:55 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 46603 invoked by uid 500); 3 Apr 2014 09:49:52 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 46596 invoked by uid 99); 3 Apr 2014 09:49:51 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Apr 2014 09:49:51 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gwang@gopivotal.com designates 209.85.192.43 as permitted sender) Received: from [209.85.192.43] (HELO mail-qg0-f43.google.com) (209.85.192.43) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Apr 2014 09:49:46 +0000 Received: by mail-qg0-f43.google.com with SMTP id f51so1490455qge.16 for ; Thu, 03 Apr 2014 02:49:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=HjWqAOMisfxykQn5uOam+iG4xf47c9gPn0m3IMGRq7M=; b=clUmAMqJHb8URsQ+PR1OT+pTP06BQwnWqaJYr/qR750uzPza0xdKwYg6/apUWgLN5v zRbkvneLlfscv0KBWHeHjRC5lf3GCPG2mZJ3bBnU5pnoAbCTsBLUfTE7vejm2wFEelO5 wyzu+DAqLvi1SGkxJ87psKhakFLmsZTxZn6sb9hWNwAbB2bZPfPOG5wU2cQIsL0KilJO eLHTgo8mz5HWpTI/REfbVs80p+/B4yKwHyna7XhcTxpYwY4qg2FCBhYCo1/CCCHJ4HqQ Zj7JsL3pMCSrKNRwP5ikiV7PjvZoLhwtSEzmoa8bFg2aRAA5Zj0uDhMHirvjjF0JRWt2 UJLw== X-Gm-Message-State: ALoCoQnIZ0r67SAwR3QokWV0IklyMtEl6LvXHy7HxZg71RWtLe/Ol764/gAW/pHeWhYaEaZmRe2L X-Received: by 10.140.29.130 with SMTP id b2mr6002947qgb.48.1396518566015; Thu, 03 Apr 2014 02:49:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.106.132 with HTTP; Thu, 3 Apr 2014 02:49:05 -0700 (PDT) In-Reply-To: References: From: Gordon Wang Date: Thu, 3 Apr 2014 17:49:05 +0800 Message-ID: Subject: Re: how to solve reducer memory problem? To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a113ab2f60cd39604f6205146 X-Virus-Checked: Checked by ClamAV on apache.org --001a113ab2f60cd39604f6205146 Content-Type: text/plain; charset=ISO-8859-1 What is the work in reducer ? Do you have any memory intensive work in reducer(eg. cache a lot of data in memory) ? I guess the OOM error comes from your code in reducer. On Thu, Apr 3, 2014 at 5:10 PM, Li Li wrote: > *mapred.child.java.opts=-Xmx2g* > > > On Thu, Apr 3, 2014 at 5:10 PM, Li Li wrote: > >> 2g >> >> >> On Thu, Apr 3, 2014 at 1:30 PM, Stanley Shi wrote: >> >>> This doesn't seem like related with the data size. >>> >>> How much memory do you use for the reducer? >>> >>> Regards, >>> *Stanley Shi,* >>> >>> >>> >>> On Thu, Apr 3, 2014 at 8:04 AM, Li Li wrote: >>> >>>> I have a map reduce program that do some matrix operations. in the >>>> reducer, it will average many large matrix(each matrix takes up >>>> 400+MB(said by Map output bytes). so if there 50 matrix to a reducer, >>>> then the total memory usage is 20GB. so the reduce task got exception: >>>> >>>> FATAL org.apache.hadoop.mapred.Child: Error running child : >>>> java.lang.OutOfMemoryError: Java heap space >>>> at org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:344) >>>> at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:406) >>>> at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:238) >>>> at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:438) >>>> at org.apache.hadoop.mapred.Merger.merge(Merger.java:142) >>>> at >>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier.createKVIterator(ReduceTask.java:2539) >>>> at >>>> org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access$400(ReduceTask.java:661) >>>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399) >>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) >>>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>>> >>>> one method I can come up with is use Combiner to save sums of some >>>> matrixs and their count >>>> but it still can solve the problem because the combiner is not fully >>>> controled by me. >>>> >>> >>> >> > -- Regards Gordon Wang --001a113ab2f60cd39604f6205146 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
What is the work in reducer ?
Do you have any memory i= ntensive work in reducer(eg. cache a lot of data in memory) ? I guess the O= OM error comes from your code in reducer. =A0


On Thu, Apr 3, 2014 at 5:10 PM, Li Li <fancyerii@gmail.com> wrote:
mapred.child.java.opts=3D-Xmx2g


On Thu, Apr 3, 2014 at 5:10 PM, Li Li <fancyerii@gmail.com&g= t; wrote:
2g


On Thu, Apr 3, 2014 at= 1:30 PM, Stanley Shi <sshi@gopivotal.com> wrote:
This doesn't seem like = related with the data size.

How much memory do you use f= or the reducer?=A0

Regards,
Stanley Shi,



On Thu, Apr 3, 2014 at 8:04 AM, Li Li <fancyerii@gmail.com> wrote:
I have a map reduce program that do some matrix operations. in the
reducer, it will average many large matrix(each matrix takes up
400+MB(said by Map output bytes). so if there 50 matrix to a reducer,
then the total memory usage is 20GB. so the reduce task got exception:

FATAL org.apache.hadoop.mapred.Child: Error running child :
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:344)
at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:406)
at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:238)
at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:438)
at org.apache.hadoop.mapred.Merger.merge(Merger.java:142)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.createKVIterator(Reduce= Task.java:2539)
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access$400(ReduceTask.j= ava:661)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:399)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformatio= n.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

one method I can come up with is use Combiner to save sums of some
matrixs and their count
but it still can solve the problem because the combiner is not fully
controled by me.






--
=
Regards
Gordon Wang
--001a113ab2f60cd39604f6205146--