Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A41E910F70 for ; Fri, 12 Jul 2013 20:51:57 +0000 (UTC) Received: (qmail 71889 invoked by uid 500); 12 Jul 2013 20:51:53 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 71790 invoked by uid 500); 12 Jul 2013 20:51:52 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 71783 invoked by uid 99); 12 Jul 2013 20:51:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Jul 2013 20:51:52 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shahab.yunus@gmail.com designates 209.85.214.51 as permitted sender) Received: from [209.85.214.51] (HELO mail-bk0-f51.google.com) (209.85.214.51) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Jul 2013 20:51:47 +0000 Received: by mail-bk0-f51.google.com with SMTP id ji1so3940430bkc.38 for ; Fri, 12 Jul 2013 13:51:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=xIgbjdizD3d7TNXe0oXm82L0awBGR/cRr6TV1oCkm/M=; b=UH0+8h0PEx2zDLDS0eLcD5Vvc//grSXvchwx1w0E/bYJUp2V7WeL4MFIxf/JvkEVVo hZy3HcIX6JL+K9qPTaY0TzukrYX1r05ePJM7YwUdu54CkUQ3yHsPmQamvt9DYXx2rXNp rJhFc/wGycwfEjP+tHPgk5HQHpL8zOLP+O1ACSsCBk9C48+MTBCHOGam3RdKBswn1qt5 RVVPRNRuK0pqBmaZFqv7hM3dGIqGJ0nk+BqZXKuJcyFeqKHcMUixs97hlUreZ2XaUL9i +xmtkTa5goQGA4KTnJmXczw3ZHKz1GFvIozdbHNSXTUFwIy3iHtgqYwC4qG2r3S9kqa5 pM/Q== MIME-Version: 1.0 X-Received: by 10.204.231.137 with SMTP id jq9mr6689021bkb.150.1373662286393; Fri, 12 Jul 2013 13:51:26 -0700 (PDT) Received: by 10.204.188.71 with HTTP; Fri, 12 Jul 2013 13:51:26 -0700 (PDT) In-Reply-To: References: Date: Fri, 12 Jul 2013 16:51:26 -0400 Message-ID: Subject: Re: How are 'PHYSICAL_MEMORY_BYTES' and 'VIRTUAL_MEMORY_BYTES' calculated? From: Shahab Yunus To: "user@hadoop.apache.org" Content-Type: multipart/alternative; boundary=485b3979db809f728c04e156ac33 X-Virus-Checked: Checked by ClamAV on apache.org --485b3979db809f728c04e156ac33 Content-Type: text/plain; charset=ISO-8859-1 As Vinod Kumar Vavilapalli they are indeed snapshots in point and time. So they are neither the peak usage from the whole duration of the job, nor cumulative aggregate that increases over time. Regards, Shahab On Fri, Jul 12, 2013 at 4:47 PM, hadoop qi wrote: > Thanks for the response. So they represent the total physical memory > (virtual memory) has been allocated to the job (e.g., from heap and stack) > during its entire life time? I am still confused how to get the cumulative > number from /proc/meminfo. I think from /proc/meminfo we can only get the > memory usage of a process in a particular time point (looked like a > snapshot of the status of the process). If these numbers are added, the sum > would be much more than memory allocated to the program. > > > On Fri, Jul 12, 2013 at 6:47 AM, Shahab Yunus wrote: > >> I think they are cumulative but per task. >> >> Physical memory bytes >> (PHYSICAL_MEMORY_BYTES) >> The physical memory being used by a task in bytes, as reported by >> /proc/meminfo. >> Virtual memory bytes >> (VIRTUAL_MEMORY_BYTES) >> The virtual memory being used by a task in bytes, as reported by >> /proc/meminfo. >> >> This is from the Definitive Guide book. Page 260. >> >> Regards, >> Shhab >> >> >> On Thu, Jul 11, 2013 at 12:47 PM, hadoop qi wrote: >> >>> Hello, >>> >>> I am wondering how memory counters 'PHYSICAL_MEMORY_BYTES' and >>> 'VIRTUAL_MEMORY_BYTES' are calculated? They are peaks of memory usage or >>> cumulative usage? >>> >>> Thanks for help, >>> >> >> > --485b3979db809f728c04e156ac33 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
As Vinod Kumar Vavilapalli they are indeed snapshots in po= int and time. So they are neither the peak usage from the whole duration of= the job, nor cumulative aggregate that increases over time.

Regards,
Shahab

On Fri, Jul 12, 2013 at 4:47 PM, hadoop qi <hadoop.jqi@gmail.com> wrote:
Thanks for the response. So= they represent the total physical memory (virtual memory) has been allocat= ed to the job (e.g., from heap and stack) during its entire life time? I am= still confused how to get the cumulative number from /proc/meminfo. I thin= k from /proc/meminfo we can only get the memory usage of a=A0 process in a = particular time point (looked like a snapshot of the status of the process)= . If these numbers are added, the sum would be much more than memory alloca= ted to the program.

On Fri, Jul 12, 2013 at 6:47 AM, Shahab Yu= nus <shahab.yunus@gmail.com> wrote:
I think they are cumul= ative but per task.

Physical memory bytes
(PHYSICAL_MEMORY_BYTES)
The physical memory being used by a task = in bytes, as reported by /proc/meminfo.
Virtual memory bytes
(VIRTUAL_MEMORY_BYTES)
The vi= rtual memory being used by a task in bytes, as reported by /proc/meminfo.

This is from the Definitive Guide book. Page 260.

Regards,
Shhab


On Thu, Jul 11, 2013 a= t 12:47 PM, hadoop qi <hadoop.jqi@gmail.com> wrote:
Hello,
<= br>
I am wondering how memory counters=A0 'PHYSICAL_MEMORY_BYTES&#= 39;=A0 and 'VIRTUAL_MEMORY_BYTES'=A0 are calculated? They are peaks= of memory usage or cumulative usage?

Thanks for help,



--485b3979db809f728c04e156ac33--