Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D5C00F0D4 for ; Mon, 25 Mar 2013 06:32:34 +0000 (UTC) Received: (qmail 98920 invoked by uid 500); 25 Mar 2013 06:32:30 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 98511 invoked by uid 500); 25 Mar 2013 06:32:25 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 97772 invoked by uid 99); 25 Mar 2013 06:32:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Mar 2013 06:32:25 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of hemanty@thoughtworks.com designates 64.18.0.22 as permitted sender) Received: from [64.18.0.22] (HELO exprod5og111.obsmtp.com) (64.18.0.22) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Mar 2013 06:32:17 +0000 Received: from mail-oa0-f72.google.com ([209.85.219.72]) (using TLSv1) by exprod5ob111.postini.com ([64.18.4.12]) with SMTP ID DSNKUU/vWxx2/jhl8WBQbIA+5NqN2g+fjvmr@postini.com; Sun, 24 Mar 2013 23:31:56 PDT Received: by mail-oa0-f72.google.com with SMTP id j6so30529102oag.3 for ; Sun, 24 Mar 2013 23:31:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:x-received:in-reply-to:references:date :message-id:subject:from:to:content-type:x-gm-message-state; bh=1q1IQAFRgtXtqE6+VbClE7BfOqWnAkKkl69ttt4RCLw=; b=Dvn5CvOwEt+sj8keg0V6kz0tWELXdQGDEgM5NK0KohvjmKtuevuXK3BULBMtitp8c4 NBmcILWRv1veoEE2zw1FhMSQ6R/eW9OCrH6d3YqXVInCWr31yDudMOy1RiqlBPp/ao7S VwpHGau5+oKj1AK/9g+bNbs1JIrK466WDAG8ZR9cGbqr33wWoMbSSg3GfTUB6etKwFt0 U2LuxVCTGykJ3fJMk/fBOTgj6zXuUwQIV+nEHkINqgiFPYAU8GcQ3pGDmJdx82x4SUnp B2JelZDh0BKsmj3x6q+v058RCmzqUHeuu5Xo5zOSiH5fiiMXd0mdcTxfDy7ulDIdYLKB GLhQ== X-Received: by 10.60.11.197 with SMTP id s5mr10180939oeb.100.1364193115411; Sun, 24 Mar 2013 23:31:55 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.60.11.197 with SMTP id s5mr10180936oeb.100.1364193115326; Sun, 24 Mar 2013 23:31:55 -0700 (PDT) Received: by 10.76.154.136 with HTTP; Sun, 24 Mar 2013 23:31:55 -0700 (PDT) In-Reply-To: References: Date: Mon, 25 Mar 2013 12:01:55 +0530 Message-ID: Subject: Re: Child JVM memory allocation / Usage From: Hemanth Yamijala To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=e89a8fb202ea0b665404d8b9f62c X-Gm-Message-State: ALoCoQmMEbStzHuhjJw9XgnBoTi1131Jx1zFWzOlWBn16yCtMG3BQ0xsGmpqSY+bvH0sJYVWRIsVPAt8hfxf85L2evzoMKKDzIKf1oy/+4GVjl5c/VaDg7YepKQgUEpKJQWDkIgC9kJKXhvtVHUnCLBVLI8ueaQMJw== X-Virus-Checked: Checked by ClamAV on apache.org --e89a8fb202ea0b665404d8b9f62c Content-Type: text/plain; charset=ISO-8859-1 Hi, The free memory might be low, just because GC hasn't reclaimed what it can. Can you just try reading in the data you want to read and see if that works ? Thanks Hemanth On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarlapudi < nagarjuna.kanamarlapudi@gmail.com> wrote: > io.sort.mb = 256 MB > > > On Monday, March 25, 2013, Harsh J wrote: > >> The MapTask may consume some memory of its own as well. What is your >> io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to? >> >> On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi >> wrote: >> > Hi, >> > >> > I configured my child jvm heap to 2 GB. So, I thought I could really >> read >> > 1.5GB of data and store it in memory (mapper/reducer). >> > >> > I wanted to confirm the same and wrote the following piece of code in >> the >> > configure method of mapper. >> > >> > @Override >> > >> > public void configure(JobConf job) { >> > >> > System.out.println("FREE MEMORY -- " >> > >> > + Runtime.getRuntime().freeMemory()); >> > >> > System.out.println("MAX MEMORY ---" + Runtime.getRuntime().maxMemory()); >> > >> > } >> > >> > >> > Surprisingly the output was >> > >> > >> > FREE MEMORY -- 341854864 = 320 MB >> > MAX MEMORY ---1908932608 = 1.9 GB >> > >> > >> > I am just wondering what processes are taking up that extra 1.6GB of >> heap >> > which I configured for the child jvm heap. >> > >> > >> > Appreciate in helping me understand the scenario. >> > >> > >> > >> > Regards >> > >> > Nagarjuna K >> > >> > >> > >> >> >> >> -- >> Harsh J >> > > > -- > Sent from iPhone > --e89a8fb202ea0b665404d8b9f62c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

The free memory might be low,= just because GC hasn't reclaimed what it can. Can you just try reading= in the data you want to read and see if that works ?

Thanks
Hemanth


On Mon, Mar 25, 2013 at 10:32= AM, nagarjuna kanamarlapudi <nagarjuna.kanamarlapudi@gmai= l.com> wrote:
io.sort.mb =3D=A0256 MB


On Monday, March 25, 2013, Harsh J wrote:
The MapTask may consume some memory of its own as well. What is your
io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to?

On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi
<nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> I configured =A0my child jvm heap to 2 GB. So, I thought I could reall= y read
> 1.5GB of data and store it in memory (mapper/reducer).
>
> I wanted to confirm the same and wrote the following piece of code in = the
> configure method of mapper.
>
> @Override
>
> public void configure(JobConf job) {
>
> System.out.println("FREE MEMORY -- "
>
> + Runtime.getRuntime().freeMemory());
>
> System.out.println("MAX MEMORY ---" + Runtime.getRuntime().m= axMemory());
>
> }
>
>
> Surprisingly the output was
>
>
> FREE MEMORY -- 341854864 =A0=3D 320 MB
> MAX MEMORY ---1908932608 =A0=3D 1.9 GB
>
>
> I am just wondering what processes are taking up that extra 1.6GB of h= eap
> which I configured for the child jvm heap.
>
>
> Appreciate in helping me understand the scenario.
>
>
>
> Regards
>
> Nagarjuna K
>
>
>



--
Harsh J


--
Sent from iPhone

--e89a8fb202ea0b665404d8b9f62c--