Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6CADEF44B for ; Mon, 25 Mar 2013 13:13:56 +0000 (UTC) Received: (qmail 83835 invoked by uid 500); 25 Mar 2013 13:13:50 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 81856 invoked by uid 500); 25 Mar 2013 13:13:48 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 81843 invoked by uid 99); 25 Mar 2013 13:13:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Mar 2013 13:13:47 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nagarjuna.kanamarlapudi@gmail.com designates 209.85.128.177 as permitted sender) Received: from [209.85.128.177] (HELO mail-ve0-f177.google.com) (209.85.128.177) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Mar 2013 13:13:42 +0000 Received: by mail-ve0-f177.google.com with SMTP id jw11so797301veb.36 for ; Mon, 25 Mar 2013 06:13:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:x-mailer:date:message-id:in-reply-to :references:x-orchestra-oid:x-orchestra-sig:x-orchestra-thrid :x-orchestra-thrid-sig:x-orchestra-account:from:to:cc:subject :content-type; bh=FqgHmfGQVWAnkl+MCCSEVStYJpJz3HNR37VoMpd0bMw=; b=Uszfdde+6X+gtq5/HHXCzK8lpfywuM4qOQ/sGdAMpXspsc60g3uTBh5hM3UE34OvzS rG5umeS7jXeTRvGkpf8so0D9oaKnzSia4kE/SHF+b86CyFHtrDWTq0DQSwxI7G5zOBic f7qRwQGdKGz82JqClkgnTvryOXaznVigCc8IYnje7b2yt2+IwXr/5g9xlT4Qx/VO0G2g X9GBxCtURl+mtHIlLZaQMuFAn4HMeo0r5lK21BHPX+IxnDuPBhn0pS2yIZ/xRRxVnFS2 g6XUBYvstaBnwMXst+ouKzsUVHpPq4F0WQrSrggeKBkY3wug4i25gq2mxR87j1VdkH9W 6FPg== X-Received: by 10.52.21.212 with SMTP id x20mr12821051vde.106.1364217201397; Mon, 25 Mar 2013 06:13:21 -0700 (PDT) Received: from [127.0.0.1] (ec2-54-235-159-149.compute-1.amazonaws.com. [54.235.159.149]) by mx.google.com with ESMTPS id i17sm20341886vdj.1.2013.03.25.06.13.20 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 25 Mar 2013 06:13:20 -0700 (PDT) MIME-Version: 1.0 X-Mailer: Nodemailer (0.3.28; +http://andris9.github.com/Nodemailer/) Date: Mon, 25 Mar 2013 06:13:20 -0700 (PDT) Message-Id: <1364217199998.556d641a@Nodemailer> In-Reply-To: References: X-Orchestra-Oid: 24FD94B7-B51D-458D-9F2D-CAD122DE2139 X-Orchestra-Sig: ea75226d48339a448af83e7dbf941fea2386e875 X-Orchestra-Thrid: T2517148B-0D8B-4877-97A4-FCD7B35FBE4C_1430383339489904965 X-Orchestra-Thrid-Sig: 51be2a25e8093b8bc2de2ea0ca3d1c318bfcbcb4 X-Orchestra-Account: 6cf2a671bac59697b44cbc519a23c4ba56839a61 From: "Nagarjuna Kanamarlapudi" To: user@hadoop.apache.org Cc: user@hadoop.apache.org Subject: Re: Child JVM memory allocation / Usage Content-Type: multipart/alternative; boundary="----mailcomposer-?=_1-1364217200719" X-Virus-Checked: Checked by ClamAV on apache.org ------mailcomposer-?=_1-1364217200719 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I have a lookup file which I need in the mapper. So I am trying to read the= whole file and load it into list in the mapper.=C2=A0 For each and every record Iook in this file which I got from distributed = cache.=C2=A0 =E2=80=94 Sent from iPhone On Mon, Mar 25, 2013 at 6:39 PM, Hemanth Yamijala wrote: > Hmm. How are you loading the file into memory =3F Is it some sort of = memory > mapping etc =3F Are they being read as records =3F Some details of the = app will > help > On Mon, Mar 25, 2013 at 2:14 PM, nagarjuna kanamarlapudi < > nagarjuna.kanamarlapudi@gmail.com> wrote: >> Hi Hemanth, >> >> I tried out your suggestion loading 420 MB file into memory. It threw = java >> heap space error. >> >> I am not sure where this 1.6 GB of configured heap went to =3F >> >> >> On Mon, Mar 25, 2013 at 12:01 PM, Hemanth Yamijala < >> yhemanth@thoughtworks.com> wrote: >> >>> Hi, >>> >>> The free memory might be low, just because GC hasn't reclaimed what it >>> can. Can you just try reading in the data you want to read and see if = that >>> works =3F >>> >>> Thanks >>> Hemanth >>> >>> >>> On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarlapudi < >>> nagarjuna.kanamarlapudi@gmail.com> wrote: >>> >>>> io.sort.mb =3D 256 MB >>>> >>>> >>>> On Monday, March 25, 2013, Harsh J wrote: >>>> >>>>> The MapTask may consume some memory of its own as well. What is your >>>>> io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to=3F >>>>> >>>>> On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi >>>>> wrote: >>>>> > Hi, >>>>> > >>>>> > I configured my child jvm heap to 2 GB. So, I thought I could = really >>>>> read >>>>> > 1.5GB of data and store it in memory (mapper/reducer). >>>>> > >>>>> > I wanted to confirm the same and wrote the following piece of code = in >>>>> the >>>>> > configure method of mapper. >>>>> > >>>>> > @Override >>>>> > >>>>> > public void configure(JobConf job) { >>>>> > >>>>> > System.out.println(=22FREE MEMORY -- =22 >>>>> > >>>>> > + Runtime.getRuntime().freeMemory()); >>>>> > >>>>> > System.out.println(=22MAX MEMORY ---=22 + >>>>> Runtime.getRuntime().maxMemory()); >>>>> > >>>>> > } >>>>> > >>>>> > >>>>> > Surprisingly the output was >>>>> > >>>>> > >>>>> > FREE MEMORY -- 341854864 =3D 320 MB >>>>> > MAX MEMORY ---1908932608 =3D 1.9 GB >>>>> > >>>>> > >>>>> > I am just wondering what processes are taking up that extra 1.6GB = of >>>>> heap >>>>> > which I configured for the child jvm heap. >>>>> > >>>>> > >>>>> > Appreciate in helping me understand the scenario. >>>>> > >>>>> > >>>>> > >>>>> > Regards >>>>> > >>>>> > Nagarjuna K >>>>> > >>>>> > >>>>> > >>>>> >>>>> >>>>> >>>>> -- >>>>> Harsh J >>>>> >>>> >>>> >>>> -- >>>> Sent from iPhone >>>> >>> >>> >> ------mailcomposer-?=_1-1364217200719 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable

I have a lookup file which I need in the mapper. So I am trying to read = the whole file and load it into list in the mapper.=C2=A0


For each and every record Iook in this file which I got from = distributed cache.=C2=A0

=E2=80=94
Sent from iPhone


On Mon, Mar 25, 2013 at 6:39 PM, Hemanth = Yamijala <yhemanth@thoughtworks.com> = wrote:

Hmm. How are you loading the file into memory =3F Is it = some sort of memory mapping etc =3F Are they being read as records =3F Some= details of the app will help

On Mon, Mar 25, 2013 at 2:14 PM, nagarjuna kanamarlapudi <nagarjuna.kanamarlapudi@gmail.= com> wrote:
Hi Hemanth,

I tried out your = suggestion loading 420 MB file into memory. It threw java heap space error.=

I am not sure where this 1.6 GB of configured = heap went to =3F


On Mon= , Mar 25, 2013 at 12:01 PM, Hemanth Yamijala <yhema= nth@thoughtworks.com> wrote:
Hi,=

The free memory might be low, just because GC hasn't = reclaimed what it can. Can you just try reading in the data you want to = read and see if that works =3F

Thanks
Hema= nth

On Mon, Mar 25, 2013 at 10:32 AM, = nagarjuna kanamarlapudi <nagarjuna.kanamarlapudi@gmail.com> = wrote:
io.= sort.mb =3D=C2=A0256 MB


On Monday, March 25, 2013, Harsh J wrote:
The MapTask may consume some memory of its own as well. What is your
io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to=3F

On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi
<nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> I configured =C2=A0my child jvm heap to 2 GB. So, I thought I could = really read
> 1.5GB of data and store it in memory (mapper/reducer).
>
> I wanted to confirm the same and wrote the following piece of code in = the
> configure method of mapper.
>
> @Override
>
> public void configure(JobConf job) {
>
> System.out.println(=22FREE MEMORY -- =22
>
> + Runtime.getRuntime().freeMemory());
>
> System.out.println(=22MAX MEMORY ---=22 + Runtime.getRuntime().= maxMemory());
>
> }
>
>
> Surprisingly the output was
>
>
> FREE MEMORY -- 341854864 =C2=A0=3D 320 MB
> MAX MEMORY ---1908932608 =C2=A0=3D 1.9 GB
>
>
> I am just wondering what processes are taking up that extra 1.6GB of = heap
> which I configured for the child jvm heap.
>
>
> Appreciate in helping me understand the scenario.
>
>
>
> Regards
>
> Nagarjuna K
>
>
>



--
Harsh J


--
Sent from iPhone




------mailcomposer-?=_1-1364217200719--