Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C3982FA09 for ; Mon, 25 Mar 2013 08:45:30 +0000 (UTC) Received: (qmail 56840 invoked by uid 500); 25 Mar 2013 08:45:27 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 54298 invoked by uid 500); 25 Mar 2013 08:45:15 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 54255 invoked by uid 99); 25 Mar 2013 08:45:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Mar 2013 08:45:13 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nagarjuna.kanamarlapudi@gmail.com designates 209.85.210.175 as permitted sender) Received: from [209.85.210.175] (HELO mail-ia0-f175.google.com) (209.85.210.175) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Mar 2013 08:45:07 +0000 Received: by mail-ia0-f175.google.com with SMTP id y26so5321978iab.34 for ; Mon, 25 Mar 2013 01:44:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type; bh=z43dhD+5XirBTxAz05MC0jqNYTsjGzfbK9t4EtLElqo=; b=wP9ygCPVOfeeOnO5pOtSNUdYu96YbHbtkeOPW4CiXwMG2FJYWHpu1KT1i3lJM1iTQf XuPRm5wdYQwmOzA8V9LtGEL6CkZySX9S8TXNRuUOqt7ZlY1acXIT1f4gqG1T3SoZBnYM 3JyCJ+WhZ84b2vCXqpJX44ROEyE9kK1iOOXzP1CKmskyvX24R7HYRW8rMsQ1h3gz1SKR cZHLLGaP8G2OppsWKgZy875+l+u0UlQeTbUHpdYU3baxefREbPfXA2tAfSogEqVBG1aV 3xqRfMNMaD5HcwIWtlB40N0mcD21xl+QprhlDZ2UyW2syoYOnPYv1cKQkxV4NsvTp8sH HgTw== MIME-Version: 1.0 X-Received: by 10.50.191.133 with SMTP id gy5mr11074011igc.50.1364201086737; Mon, 25 Mar 2013 01:44:46 -0700 (PDT) Received: by 10.64.96.36 with HTTP; Mon, 25 Mar 2013 01:44:46 -0700 (PDT) In-Reply-To: References: Date: Mon, 25 Mar 2013 14:14:46 +0530 Message-ID: Subject: Re: Child JVM memory allocation / Usage From: nagarjuna kanamarlapudi To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=14dae93407192d66c104d8bbd161 X-Virus-Checked: Checked by ClamAV on apache.org --14dae93407192d66c104d8bbd161 Content-Type: text/plain; charset=ISO-8859-1 Hi Hemanth, I tried out your suggestion loading 420 MB file into memory. It threw java heap space error. I am not sure where this 1.6 GB of configured heap went to ? On Mon, Mar 25, 2013 at 12:01 PM, Hemanth Yamijala < yhemanth@thoughtworks.com> wrote: > Hi, > > The free memory might be low, just because GC hasn't reclaimed what it > can. Can you just try reading in the data you want to read and see if that > works ? > > Thanks > Hemanth > > > On Mon, Mar 25, 2013 at 10:32 AM, nagarjuna kanamarlapudi < > nagarjuna.kanamarlapudi@gmail.com> wrote: > >> io.sort.mb = 256 MB >> >> >> On Monday, March 25, 2013, Harsh J wrote: >> >>> The MapTask may consume some memory of its own as well. What is your >>> io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to? >>> >>> On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi >>> wrote: >>> > Hi, >>> > >>> > I configured my child jvm heap to 2 GB. So, I thought I could really >>> read >>> > 1.5GB of data and store it in memory (mapper/reducer). >>> > >>> > I wanted to confirm the same and wrote the following piece of code in >>> the >>> > configure method of mapper. >>> > >>> > @Override >>> > >>> > public void configure(JobConf job) { >>> > >>> > System.out.println("FREE MEMORY -- " >>> > >>> > + Runtime.getRuntime().freeMemory()); >>> > >>> > System.out.println("MAX MEMORY ---" + >>> Runtime.getRuntime().maxMemory()); >>> > >>> > } >>> > >>> > >>> > Surprisingly the output was >>> > >>> > >>> > FREE MEMORY -- 341854864 = 320 MB >>> > MAX MEMORY ---1908932608 = 1.9 GB >>> > >>> > >>> > I am just wondering what processes are taking up that extra 1.6GB of >>> heap >>> > which I configured for the child jvm heap. >>> > >>> > >>> > Appreciate in helping me understand the scenario. >>> > >>> > >>> > >>> > Regards >>> > >>> > Nagarjuna K >>> > >>> > >>> > >>> >>> >>> >>> -- >>> Harsh J >>> >> >> >> -- >> Sent from iPhone >> > > --14dae93407192d66c104d8bbd161 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Hemanth,

I tried out your sugg= estion loading 420 MB file into memory. It threw java heap space error.

I am not sure where this 1.6 GB of config= ured heap went to ?


On Mon,= Mar 25, 2013 at 12:01 PM, Hemanth Yamijala <yhemanth@thoughtwor= ks.com> wrote:
Hi,

The = free memory might be low, just because GC hasn't reclaimed what it can.= Can you just try reading in the data you want to read and see if that work= s ?

Thanks
= Hemanth


On Mon, Mar 25,= 2013 at 10:32 AM, nagarjuna kanamarlapudi <nagarjuna.kana= marlapudi@gmail.com> wrote:
io.sort.mb =3D=A0256 MB


On Monday, March 25, 2013, Harsh J wrote:
The MapTask may consume some memory of its own as well. What is your
io.sort.mb (MR1) or mapreduce.task.io.sort.mb (MR2) set to?

On Sun, Mar 24, 2013 at 3:40 PM, nagarjuna kanamarlapudi
<nagarjuna.kanamarlapudi@gmail.com> wrote:
> Hi,
>
> I configured =A0my child jvm heap to 2 GB. So, I thought I could reall= y read
> 1.5GB of data and store it in memory (mapper/reducer).
>
> I wanted to confirm the same and wrote the following piece of code in = the
> configure method of mapper.
>
> @Override
>
> public void configure(JobConf job) {
>
> System.out.println("FREE MEMORY -- "
>
> + Runtime.getRuntime().freeMemory());
>
> System.out.println("MAX MEMORY ---" + Runtime.getRuntime().m= axMemory());
>
> }
>
>
> Surprisingly the output was
>
>
> FREE MEMORY -- 341854864 =A0=3D 320 MB
> MAX MEMORY ---1908932608 =A0=3D 1.9 GB
>
>
> I am just wondering what processes are taking up that extra 1.6GB of h= eap
> which I configured for the child jvm heap.
>
>
> Appreciate in helping me understand the scenario.
>
>
>
> Regards
>
> Nagarjuna K
>
>
>



--
Harsh J


--
Sent = from iPhone


--14dae93407192d66c104d8bbd161--