Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.160.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAu13zHTsXF0dNTJbtDjd7Kh7CnFdHiw=r1zXfksJ73EyWrKOA@mail.gmail.com>
References: 
 <CAAu13zHTsXF0dNTJbtDjd7Kh7CnFdHiw=r1zXfksJ73EyWrKOA@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Wed, 2 May 2012 10:06:13 +0530
Message-ID: 
 <CAOcnVr02DirYA6un0rs6tNXTd_u2KwHXpkF5rGE6Vq7+quRKig@mail.gmail.com>
Subject: Re: heap space error, low memory job, unit test
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

What is your unit test runner JVM heap size?

By default, for MR jobs the config "io.sort.mb" is "100" MB, so if it
can't allocate that many bytes into an array then it would fail. Lower
the "io.sort.mb" to "20" or so before submitting your job and that
should help you get rid of this.

On Wed, May 2, 2012 at 5:51 AM, Jay Vyas <jayunit100@gmail.com> wrote:
> Hi guys :
>
>
> I have a map/r job that has always worked fine, but which fails due to a
> heap space error on my local machine during unit tests.
>
> It runs in hadoop's default mode, and just fails durring the constructor =
of
> the MapOutputBuffer.... Any thoughts on why ?
>
> I dont do any custom memory settings in by unit tests, because they aren'=
t
> really needed --- So I assume this is related to /tmp files
> or something ... but cant track down the issue.
>
> Any thoughts would be very much appreciated ..
>
> 12/05/01 19:15:53 WARN mapred.LocalJobRunner: job_local_0002
> java.lang.OutOfMemoryError: Java heap space
> =A0 =A0at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:807)
> =A0 =A0at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:5=
57)
>
>
>
>
> --
> Jay Vyas
> MMSB/UCHC


--=20
Harsh J