Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B38BBF543 for ; Tue, 26 Mar 2013 15:53:46 +0000 (UTC) Received: (qmail 36669 invoked by uid 500); 26 Mar 2013 15:53:40 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 36559 invoked by uid 500); 26 Mar 2013 15:53:40 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 36552 invoked by uid 99); 26 Mar 2013 15:53:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Mar 2013 15:53:40 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of hemanty@thoughtworks.com designates 64.18.0.24 as permitted sender) Received: from [64.18.0.24] (HELO exprod5og112.obsmtp.com) (64.18.0.24) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Mar 2013 15:53:33 +0000 Received: from mail-oa0-f71.google.com ([209.85.219.71]) (using TLSv1) by exprod5ob112.postini.com ([64.18.4.12]) with SMTP ID DSNKUVHEZyxJbEdXhDa8iFG1/Eant2+m5EWk@postini.com; Tue, 26 Mar 2013 08:53:12 PDT Received: by mail-oa0-f71.google.com with SMTP id o6so38997424oag.2 for ; Tue, 26 Mar 2013 08:53:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-received:in-reply-to:references:date :message-id:subject:from:to:content-type:x-gm-message-state; bh=31D9n++A30iQZDaVRQW+mWMobVbOuxJgwEDbZ1xw00c=; b=gUOY9FSCKoe4W2hG8QvC2UIcdN/KitsYhHDToYjygW6kJQloTOqEXkNotI/X+Gbd1/ mAvD/wBu6CeYnFQFLtxnEkfCb5yMj1jjpTumedpHc3K09YIhU24YtctkXn/g0AA7bQbV p97c1SqKLqXvJdtAk+kB4iSHLLkdNWQ9t7ebLcrvbHUIqHnOt4VVSjqGBc8vjsNNrxgT 5980xTe3I4BRnZV69KSw/2tpuMTUGm/0OtqjJbbQQnuQfX3MYKvqMk1r3p6qwAPmXh0g UNgfPL1Uhj36QM+/2tq47it8LSSv+W07+r7GZKW1UbC3uYbSLb99eF3dJMfUnPWrQfBK XQjA== X-Received: by 10.182.72.5 with SMTP id z5mr2345824obu.24.1364313189521; Tue, 26 Mar 2013 08:53:09 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.182.72.5 with SMTP id z5mr2345818obu.24.1364313189348; Tue, 26 Mar 2013 08:53:09 -0700 (PDT) Received: by 10.76.154.136 with HTTP; Tue, 26 Mar 2013 08:53:09 -0700 (PDT) In-Reply-To: References: <1364217199998.556d641a@Nodemailer> Date: Tue, 26 Mar 2013 21:23:09 +0530 Message-ID: Subject: Re: Child JVM memory allocation / Usage From: Hemanth Yamijala To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=f46d0447853303dae004d8d5ebc1 X-Gm-Message-State: ALoCoQnS39mhNAellfVt49hYqh9bySwQcuB/otuiXn5Q53eSj4DOAGbrpt6mzr8mBDIUxrabGPHDWf4XjKBfzkBwCT/W9MFpc3fHcvKOzhNhjSV/pOeiUozMDsV6Onvq3bDrCRQiovLXMPx6H9ZL/umSAL4HaRkDrA== X-Virus-Checked: Checked by ClamAV on apache.org --f46d0447853303dae004d8d5ebc1 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hi, I tried to use the -XX:+HeapDumpOnOutOfMemoryError. Unfortunately, like I suspected, the dump goes to the current work directory of the task attempt as it executes on the cluster. This directory is cleaned up once the task is done. There are options to keep failed task files or task files matching a pattern. However, these are NOT retaining the current working directory. Hence, there is no option to get this from a cluster AFAIK. You are effectively left with the jmap option on pseudo distributed cluster I think. Thanks Hemanth On Tue, Mar 26, 2013 at 11:37 AM, Hemanth Yamijala < yhemanth@thoughtworks.com> wrote: > If your task is running out of memory, you could add the option * > -XX:+HeapDumpOnOutOfMemoryError * > *to *mapred.child.java.opts (along with the heap memory). However, I am > not sure where it stores the dump.. You might need to experiment a littl= e > on it.. Will try and send out the info if I get time to try out. > > > Thanks > Hemanth > > > On Tue, Mar 26, 2013 at 10:23 AM, nagarjuna kanamarlapudi < > nagarjuna.kanamarlapudi@gmail.com> wrote: > >> Hi hemanth, >> >> This sounds interesting, will out try out that on the pseudo cluster. >> But the real problem for me is, the cluster is being maintained by thir= d >> party. I only have have a edge node through which I can submit the jobs. >> >> Is there any other way of getting the dump instead of physically going t= o >> that machine and checking out. >> >> >> >> On Tue, Mar 26, 2013 at 10:12 AM, Hemanth Yamijala < >> yhemanth@thoughtworks.com> wrote: >> >>> Hi, >>> >>> One option to find what could be taking the memory is to use jmap on th= e >>> running task. The steps I followed are: >>> >>> - I ran a sleep job (which comes in the examples jar of the distributio= n >>> - effectively does nothing in the mapper / reducer). >>> - From the JobTracker UI looked at a map task attempt ID. >>> - Then on the machine where the map task is running, got the PID of the >>> running task - ps -ef | grep >>> - On the same machine executed jmap -histo >>> >>> This will give you an idea of the count of objects allocated and size. >>> Jmap also has options to get a dump, that will contain more information= , >>> but this should help to get you started with debugging. >>> >>> For my sleep job task - I saw allocations worth roughly 130 MB. >>> >>> Thanks >>> hemanth >>> >>> >>> >>> >>> On Mon, Mar 25, 2013 at 6:43 PM, Nagarjuna Kanamarlapudi < >>> nagarjuna.kanamarlapudi@gmail.com> wrote: >>> >>>> I have a lookup file which I need in the mapper. So I am trying to rea= d >>>> the whole file and load it into list in the mapper. >>>> >>>> For each and every record Iook in this file which I got from >>>> distributed cache. >>>> >>>> =97 >>>> Sent from iPhone >>>> >>>> >>>> On Mon, Mar 25, 2013 at 6:39 PM, Hemanth Yamijala < >>>> yhemanth@thoughtworks.com> wrote: >>>> >>>>> Hmm. How are you loading the file into memory ? Is it some sort of >>>>> memory mapping etc ? Are they being read as records ? Some details of= the >>>>> app will help >>>>> >>>>> >>>>> On Mon, Mar 25, 2013 at 2:14 PM, nagarjuna kanamarlapudi < >>>>> nagarjuna.kanamarlapudi@gmail.com> wrote: >>>>> >>>>>> Hi Hemanth, >>>>>> >>>>>> I tried out your suggestion loading 420 MB file into memory. It thre= w >>>>>> java heap space error. >>>>>> >>>>>> I am not sure where this 1.6 GB of configured heap went to ? >>>>>> >>>>>> >>>>>> On Mon, Mar 25, 2013 at 12:01 PM, Hemanth Yamijala < >>>>>> yhemanth@thoughtworks.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> The free memory might be low, just because GC hasn't reclaimed what >>>>>>> it can. Can you just try reading in the data you want to read and s= ee if >>>>>>> that works ? >>>>>>> >>>>>>> Thanks >>>>>>> Hemanth >>>>>>> >>>>>>> >>>>>>> On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarlapudi < >>>>>>> nagarjuna.kanamarlapudi@gmail.com> wrote: >>>>>>> >>>>>>>> io.sort.mb =3D 256 MB >>>>>>>> >>>>>>>> >>>>>>>> On Monday, March 25, 2013, Harsh J wrote: >>>>>>>> >>>>>>>>> The MapTask may consume some memory of its own as well. What is >>>>>>>>> your >>>>>>>>> io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to? >>>>>>>>> >>>>>>>>> On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi >>>>>>>>> wrote: >>>>>>>>> > Hi, >>>>>>>>> > >>>>>>>>> > I configured my child jvm heap to 2 GB. So, I thought I could >>>>>>>>> really read >>>>>>>>> > 1.5GB of data and store it in memory (mapper/reducer). >>>>>>>>> > >>>>>>>>> > I wanted to confirm the same and wrote the following piece of >>>>>>>>> code in the >>>>>>>>> > configure method of mapper. >>>>>>>>> > >>>>>>>>> > @Override >>>>>>>>> > >>>>>>>>> > public void configure(JobConf job) { >>>>>>>>> > >>>>>>>>> > System.out.println("FREE MEMORY -- " >>>>>>>>> > >>>>>>>>> > + Runtime.getRuntime().freeMemory()); >>>>>>>>> > >>>>>>>>> > System.out.println("MAX MEMORY ---" + >>>>>>>>> Runtime.getRuntime().maxMemory()); >>>>>>>>> > >>>>>>>>> > } >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Surprisingly the output was >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > FREE MEMORY -- 341854864 =3D 320 MB >>>>>>>>> > MAX MEMORY ---1908932608 =3D 1.9 GB >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > I am just wondering what processes are taking up that extra >>>>>>>>> 1.6GB of heap >>>>>>>>> > which I configured for the child jvm heap. >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Appreciate in helping me understand the scenario. >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > Regards >>>>>>>>> > >>>>>>>>> > Nagarjuna K >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Harsh J >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Sent from iPhone >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --f46d0447853303dae004d8d5ebc1 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Hi,

I tried to use the -XX:+HeapDumpOnO= utOfMemoryError. Unfortunately, like I suspected, the dump goes to the curr= ent work directory of the task attempt as it executes on the cluster. This = directory is cleaned up once the task is done. There are options to keep fa= iled task files or task files matching a pattern. However, these are NOT re= taining the current working directory. Hence, there is no option to get thi= s from a cluster AFAIK.

You are effectively left with the jmap option on = pseudo distributed cluster I think.

Th= anks
Hemanth


On Tue, Mar 26, 2013 at 11:37 AM, Hemanth Yamija= la <yhemanth@thoughtworks.com> wrote:
If your task is running out of memory, you could add the o= ption=A0-XX:+HeapDumpOnOutOfMemoryError=A0
to=A0mapred.child.java.opts (along with the heap memory). However, = I am not sure =A0where it stores the dump.. You might need to experiment a = little on it.. Will try and send out the info if I get time to try out.

Thanks
Hemanth

<= br>
On Tue, Mar 26, 2013 at 10:23 AM, nagarjuna k= anamarlapudi <nagarjuna.kanamarlapudi@gmail.com> wrote:
Hi hemanth,

<= div>This sounds interesting, will out try out that on the pseudo cluster. = =A0But the real problem for me is, the cluster is being maintained by third= party. I only have have a edge node through which I can submit the jobs.= =A0

Is there any other way of getting the dump instead of p= hysically going to that machine and =A0checking out.=A0



On Tue, Mar 26, 2013 at 10:12 AM, Hemanth Yamijala <yhemanth@thoug= htworks.com> wrote:
Hi,

One option to find what could be ta= king the memory is to use jmap on the running task. The steps I followed ar= e:

- I ran a sleep job (which comes in the example= s jar of the distribution - effectively does nothing in the mapper / reduce= r).=A0
- From the JobTracker UI looked at a map task attempt ID.
- = Then on the machine where the map task is running, got the PID of the runni= ng task - ps -ef | grep <task attempt id>
- On the same machine executed jmap -histo <pid>

=
This will give you an idea of the count of objects allocated and size.= Jmap also has options to get a dump, that will contain more information, b= ut this should help to get you started with debugging.

For my sleep job task - I saw allocations worth roughly= 130 MB.

Thanks
hemanth




On Mon, Mar 25, 2013 at 6:43 PM, Nagarju= na Kanamarlapudi <nagarjuna.kanamarlapudi@gmail.com>= ; wrote:

I have a lookup file which I need in the = mapper. So I am trying to read the whole file and load it into list in the = mapper.=A0


For each and every record Iook in this file which I got= from distributed cache.=A0

=97
Sent from iPho= ne


On Mon, Mar 25, 2013 at 6:39 PM, Hemanth Yamijala <= yhemanth@tho= ughtworks.com> wrote:

Hmm. How are you loading the file into memory ? Is it some= sort of memory mapping etc ? Are they being read as records ? Some details= of the app will help


On Mon, Mar 25, 2013 at 2:14 PM, nagarjuna kanamarlapudi = <= nagarjuna.kanamarlapudi@gmail.com> wrote:
Hi Hemanth,

I tried out your suggestion= loading 420 MB file into memory. It threw java heap space error.

I am not sure where this 1.6 GB of configured heap went to = ?


On Mon, Mar 25, 2013 at 12:01 PM, Hemanth Yamijala <= ;yhemanth@th= oughtworks.com> wrote:
Hi,

The = free memory might be low, just because GC hasn't reclaimed what it can.= Can you just try reading in the data you want to read and see if that work= s ?

Thanks
Hemanth<= /div>


On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarla= pudi <nagarjuna.kanamarlapudi@gmail.com> wro= te:
io.sort.mb =3D=A0256 MB


On Monday, March 25, 2013, Harsh J wrote:
The MapTask may consume some memory of its own as well. What is your
io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to?

On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi
<nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> I configured =A0my child jvm heap to 2 GB. So, I thought I could reall= y read
> 1.5GB of data and store it in memory (mapper/reducer).
>
> I wanted to confirm the same and wrote the following piece of code in = the
> configure method of mapper.
>
> @Override
>
> public void configure(JobConf job) {
>
> System.out.println("FREE MEMORY -- "
>
> + Runtime.getRuntime().freeMemory());
>
> System.out.println("MAX MEMORY ---" + Runtime.getRuntime().m= axMemory());
>
> }
>
>
> Surprisingly the output was
>
>
> FREE MEMORY -- 341854864 =A0=3D 320 MB
> MAX MEMORY ---1908932608 =A0=3D 1.9 GB
>
>
> I am just wondering what processes are taking up that extra 1.6GB of h= eap
> which I configured for the child jvm heap.
>
>
> Appreciate in helping me understand the scenario.
>
>
>
> Regards
>
> Nagarjuna K
>
>
>



--
Harsh J


-= -
Sent from iPhone








--f46d0447853303dae004d8d5ebc1--