Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 393A6F1E1 for ; Tue, 26 Mar 2013 06:08:27 +0000 (UTC) Received: (qmail 64529 invoked by uid 500); 26 Mar 2013 06:08:22 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 64246 invoked by uid 500); 26 Mar 2013 06:08:19 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 64207 invoked by uid 99); 26 Mar 2013 06:08:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Mar 2013 06:08:18 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hemanty@thoughtworks.com designates 64.18.0.20 as permitted sender) Received: from [64.18.0.20] (HELO exprod5og110.obsmtp.com) (64.18.0.20) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Mar 2013 06:08:12 +0000 Received: from mail-ie0-f200.google.com ([209.85.223.200]) (using TLSv1) by exprod5ob110.postini.com ([64.18.4.12]) with SMTP ID DSNKUVE7ON381Y26dfn46ELrP0igyQpP/5KW@postini.com; Mon, 25 Mar 2013 23:07:52 PDT Received: by mail-ie0-f200.google.com with SMTP id c11so35647411ieb.11 for ; Mon, 25 Mar 2013 23:07:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-received:in-reply-to:references:date :message-id:subject:from:to:content-type:x-gm-message-state; bh=m+aP9bG49ynNgSl6QArjHFJHGNbcxK9zVcE0UUvh8wQ=; b=Mw2GbC9dOMm7DuZqunbHepxi65Ij1HzBOKGHmfm0XcK6+3uI0Y8fccN+eJ2X8xLpDR BE6RqeoqaoqhsDpXBXKDwqubq010GuqmGNkDaNvHhcJDorhYTQkcGHL4Ym6UU1IZmPs5 3ZFa3pjr5wRPvye8wOjWmUB78BFj7nqdsFTW9vpF6C9v/Z3P1bIN6XZYnvVkqytFk/PA SLdJETPDgTSsjsBJ0Ea1iClf0k7/Sjscu+h/kd3trbzMDfwJfFZr6u+ngwsIZVPgGw/d 2pShxKsLPXrILR0sSnTaMLCeiLdkMMhw4PRCgQv4V/6OlDEehJ302mr5aS/glqa+fcF1 pUlQ== X-Received: by 10.182.119.72 with SMTP id ks8mr1625799obb.23.1364278071840; Mon, 25 Mar 2013 23:07:51 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.182.119.72 with SMTP id ks8mr1625797obb.23.1364278071695; Mon, 25 Mar 2013 23:07:51 -0700 (PDT) Received: by 10.76.154.136 with HTTP; Mon, 25 Mar 2013 23:07:51 -0700 (PDT) In-Reply-To: References: <1364217199998.556d641a@Nodemailer> Date: Tue, 26 Mar 2013 11:37:51 +0530 Message-ID: Subject: Re: Child JVM memory allocation / Usage From: Hemanth Yamijala To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=f46d04451a01d6fcd804d8cdbd4e X-Gm-Message-State: ALoCoQnCGCT83FBLY+bFrlPBMVTgZ8Z6stUPwfuYiGdNvUHh4HHyUTMZCmJbnR+SJwK+dvrYxmF3lhqN523XVXynYZHOiz8q2LaNWDzIDmp38UWJ7qmtNjp8qSYHHia3oUNNrHeeDjT5QOpchGnhCXlKT9xcYRsQjQ== X-Virus-Checked: Checked by ClamAV on apache.org --f46d04451a01d6fcd804d8cdbd4e Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable If your task is running out of memory, you could add the option * -XX:+HeapDumpOnOutOfMemoryError * *to *mapred.child.java.opts (along with the heap memory). However, I am not sure where it stores the dump.. You might need to experiment a little on it.. Will try and send out the info if I get time to try out. Thanks Hemanth On Tue, Mar 26, 2013 at 10:23 AM, nagarjuna kanamarlapudi < nagarjuna.kanamarlapudi@gmail.com> wrote: > Hi hemanth, > > This sounds interesting, will out try out that on the pseudo cluster. Bu= t > the real problem for me is, the cluster is being maintained by third part= y. > I only have have a edge node through which I can submit the jobs. > > Is there any other way of getting the dump instead of physically going to > that machine and checking out. > > > > On Tue, Mar 26, 2013 at 10:12 AM, Hemanth Yamijala < > yhemanth@thoughtworks.com> wrote: > >> Hi, >> >> One option to find what could be taking the memory is to use jmap on the >> running task. The steps I followed are: >> >> - I ran a sleep job (which comes in the examples jar of the distribution >> - effectively does nothing in the mapper / reducer). >> - From the JobTracker UI looked at a map task attempt ID. >> - Then on the machine where the map task is running, got the PID of the >> running task - ps -ef | grep >> - On the same machine executed jmap -histo >> >> This will give you an idea of the count of objects allocated and size. >> Jmap also has options to get a dump, that will contain more information, >> but this should help to get you started with debugging. >> >> For my sleep job task - I saw allocations worth roughly 130 MB. >> >> Thanks >> hemanth >> >> >> >> >> On Mon, Mar 25, 2013 at 6:43 PM, Nagarjuna Kanamarlapudi < >> nagarjuna.kanamarlapudi@gmail.com> wrote: >> >>> I have a lookup file which I need in the mapper. So I am trying to read >>> the whole file and load it into list in the mapper. >>> >>> For each and every record Iook in this file which I got from distribute= d >>> cache. >>> >>> =97 >>> Sent from iPhone >>> >>> >>> On Mon, Mar 25, 2013 at 6:39 PM, Hemanth Yamijala < >>> yhemanth@thoughtworks.com> wrote: >>> >>>> Hmm. How are you loading the file into memory ? Is it some sort of >>>> memory mapping etc ? Are they being read as records ? Some details of = the >>>> app will help >>>> >>>> >>>> On Mon, Mar 25, 2013 at 2:14 PM, nagarjuna kanamarlapudi < >>>> nagarjuna.kanamarlapudi@gmail.com> wrote: >>>> >>>>> Hi Hemanth, >>>>> >>>>> I tried out your suggestion loading 420 MB file into memory. It threw >>>>> java heap space error. >>>>> >>>>> I am not sure where this 1.6 GB of configured heap went to ? >>>>> >>>>> >>>>> On Mon, Mar 25, 2013 at 12:01 PM, Hemanth Yamijala < >>>>> yhemanth@thoughtworks.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> The free memory might be low, just because GC hasn't reclaimed what >>>>>> it can. Can you just try reading in the data you want to read and se= e if >>>>>> that works ? >>>>>> >>>>>> Thanks >>>>>> Hemanth >>>>>> >>>>>> >>>>>> On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarlapudi < >>>>>> nagarjuna.kanamarlapudi@gmail.com> wrote: >>>>>> >>>>>>> io.sort.mb =3D 256 MB >>>>>>> >>>>>>> >>>>>>> On Monday, March 25, 2013, Harsh J wrote: >>>>>>> >>>>>>>> The MapTask may consume some memory of its own as well. What is yo= ur >>>>>>>> io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to? >>>>>>>> >>>>>>>> On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi >>>>>>>> wrote: >>>>>>>> > Hi, >>>>>>>> > >>>>>>>> > I configured my child jvm heap to 2 GB. So, I thought I could >>>>>>>> really read >>>>>>>> > 1.5GB of data and store it in memory (mapper/reducer). >>>>>>>> > >>>>>>>> > I wanted to confirm the same and wrote the following piece of >>>>>>>> code in the >>>>>>>> > configure method of mapper. >>>>>>>> > >>>>>>>> > @Override >>>>>>>> > >>>>>>>> > public void configure(JobConf job) { >>>>>>>> > >>>>>>>> > System.out.println("FREE MEMORY -- " >>>>>>>> > >>>>>>>> > + Runtime.getRuntime().freeMemory()); >>>>>>>> > >>>>>>>> > System.out.println("MAX MEMORY ---" + >>>>>>>> Runtime.getRuntime().maxMemory()); >>>>>>>> > >>>>>>>> > } >>>>>>>> > >>>>>>>> > >>>>>>>> > Surprisingly the output was >>>>>>>> > >>>>>>>> > >>>>>>>> > FREE MEMORY -- 341854864 =3D 320 MB >>>>>>>> > MAX MEMORY ---1908932608 =3D 1.9 GB >>>>>>>> > >>>>>>>> > >>>>>>>> > I am just wondering what processes are taking up that extra 1.6G= B >>>>>>>> of heap >>>>>>>> > which I configured for the child jvm heap. >>>>>>>> > >>>>>>>> > >>>>>>>> > Appreciate in helping me understand the scenario. >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > Regards >>>>>>>> > >>>>>>>> > Nagarjuna K >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Harsh J >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sent from iPhone >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > --f46d04451a01d6fcd804d8cdbd4e Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
If your task is running out of memory, you could add the o= ption=A0-XX:+HeapDumpOnOutOfMemoryError=A0
to=A0mapred.child.java.opts (along with the heap memory). How= ever, I am not sure =A0where it stores the dump.. You might need to experim= ent a little on it.. Will try and send out the info if I get time to try ou= t.

=

Thank= s
Hemanth


On Tue, Mar 26, 20= 13 at 10:23 AM, nagarjuna kanamarlapudi <nagarjuna.kanamar= lapudi@gmail.com> wrote:
Hi hemanth,

<= div>This sounds interesting, will out try out that on the pseudo cluster. = =A0But the real problem for me is, the cluster is being maintained by third= party. I only have have a edge node through which I can submit the jobs.= =A0

Is there any other way of getting the dump instead of p= hysically going to that machine and =A0checking out.=A0

<= br>
On Tue, Mar 26, 2013 at 10:12 AM, Hemanth Yamijala <yhemanth@thoug= htworks.com> wrote:
Hi,

One option to find what could be ta= king the memory is to use jmap on the running task. The steps I followed ar= e:

- I ran a sleep job (which comes in the example= s jar of the distribution - effectively does nothing in the mapper / reduce= r).=A0
- From the JobTracker UI looked at a map task attempt ID.
- = Then on the machine where the map task is running, got the PID of the runni= ng task - ps -ef | grep <task attempt id>
- On the same machine executed jmap -histo <pid>

=
This will give you an idea of the count of objects allocated and size.= Jmap also has options to get a dump, that will contain more information, b= ut this should help to get you started with debugging.

For my sleep job task - I saw allocations worth roughly= 130 MB.

Thanks
hemanth




On Mon, Mar 25, 2013 at 6:43 PM, Nagarju= na Kanamarlapudi <nagarjuna.kanamarlapudi@gmail.com>= ; wrote:

I have a lookup file which I need in the = mapper. So I am trying to read the whole file and load it into list in the = mapper.=A0


For each and every record Iook in this file which I got= from distributed cache.=A0

=97
Sent from iPho= ne


On Mon, Mar 25, 2013 at 6:39 PM, Hemanth Yamijala <= yhemanth@tho= ughtworks.com> wrote:

Hmm. How are you loading the file into memory ? Is it some= sort of memory mapping etc ? Are they being read as records ? Some details= of the app will help


On Mon, Mar 25, 2013 at 2:14 PM, nagarjuna kanamarlapudi = <= nagarjuna.kanamarlapudi@gmail.com> wrote:
Hi Hemanth,

I tried out your suggestion= loading 420 MB file into memory. It threw java heap space error.

I am not sure where this 1.6 GB of configured heap went to = ?


On Mon, Mar 25, 2013 at 12:01 PM, Hemanth Yamijala <= ;yhemanth@th= oughtworks.com> wrote:
Hi,

The = free memory might be low, just because GC hasn't reclaimed what it can.= Can you just try reading in the data you want to read and see if that work= s ?

Thanks
Hemanth<= /div>


On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarla= pudi <nagarjuna.kanamarlapudi@gmail.com> wro= te:
io.sort.mb =3D=A0256 MB


On Monday, March 25, 2013, Harsh J wrote:
The MapTask may consume some memory of its own as well. What is your
io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to?

On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi
<nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> I configured =A0my child jvm heap to 2 GB. So, I thought I could reall= y read
> 1.5GB of data and store it in memory (mapper/reducer).
>
> I wanted to confirm the same and wrote the following piece of code in = the
> configure method of mapper.
>
> @Override
>
> public void configure(JobConf job) {
>
> System.out.println("FREE MEMORY -- "
>
> + Runtime.getRuntime().freeMemory());
>
> System.out.println("MAX MEMORY ---" + Runtime.getRuntime().m= axMemory());
>
> }
>
>
> Surprisingly the output was
>
>
> FREE MEMORY -- 341854864 =A0=3D 320 MB
> MAX MEMORY ---1908932608 =A0=3D 1.9 GB
>
>
> I am just wondering what processes are taking up that extra 1.6GB of h= eap
> which I configured for the child jvm heap.
>
>
> Appreciate in helping me understand the scenario.
>
>
>
> Regards
>
> Nagarjuna K
>
>
>



--
Harsh J


-= -
Sent from iPhone







--f46d04451a01d6fcd804d8cdbd4e--