Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7B79B10ACE for ; Sun, 22 Feb 2015 06:22:49 +0000 (UTC) Received: (qmail 64146 invoked by uid 500); 22 Feb 2015 06:22:42 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 64031 invoked by uid 500); 22 Feb 2015 06:22:42 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 64021 invoked by uid 99); 22 Feb 2015 06:22:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Feb 2015 06:22:42 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [176.58.122.69] (HELO mail.eagleeyet.net) (176.58.122.69) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 22 Feb 2015 06:22:37 +0000 Received: by mail.eagleeyet.net (Postfix, from userid 33) id EA2CE1853B; Sun, 22 Feb 2015 06:22:13 +0000 (UTC) To: user@hadoop.apache.org Subject: Re: How can I get the memory usage in Namenode and =?UTF-8?Q?Datanode=3F?= X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="=_f0ab0635288ad0c6e205248c5d4c83f6" Date: Sun, 22 Feb 2015 07:22:13 +0100 From: Jonathan Aquilina Cc: Fang Zhou In-Reply-To: <671CF7F9-B54E-404C-A519-E8878218BF34@gmail.com> References: <671CF7F9-B54E-404C-A519-E8878218BF34@gmail.com> Message-ID: <5f4995bc7faa958e0de13968bdf3200d@mail.eagleeyet.net> X-Sender: jaquilina@eagleeyet.net User-Agent: Roundcube Webmail/1.0.5 X-Virus-Checked: Checked by ClamAV on apache.org --=_f0ab0635288ad0c6e205248c5d4c83f6 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII I am rather new to hadoop, but wouldnt the difference be potentially in how the files are split in terms of size? --- Regards, Jonathan Aquilina Founder Eagle Eye T On 2015-02-21 21:54, Fang Zhou wrote: > Hi All, > > I want to test the memory usage on Namenode and Datanode. > > I try to use jmap, jstat, proc/pid/stat, top, ps aux, and Hadoop website interface to check the memory. > The values I get from them are different. I also found that the memory always changes periodically. > This is the first thing confused me. > > I thought the more files stored in Namenode, the more memory usage in Namenode and Datanode. > I also thought the memory used in Namenode should be larger than the memory used in each Datanode. > However, some results show my ideas are wrong. > For example, I test the memory usage of Namenode with 6000 and 1000 files. > The "6000" memory is less than "1000" memory from jmap's results. > I also found that the memory usage in Datanode is larger than the memory used in Namenode. > > I really don't know how to get the memory usage in Namenode and Datanode. > > Can anyone give me some advices? > > Thanks, > Tim --=_f0ab0635288ad0c6e205248c5d4c83f6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8

I am rather new to hadoop, but wouldnt the difference be potentially in = how the files are split in terms of size?

---
Regards,
Jonathan Aquilina
Founder Eagle Eye T

On 2015-02-21 21:54, Fang Zhou wrote:

Hi All,

I want to test the memory usage on Namenode and Datanode.

I try to use jmap, jstat, proc/pid/stat, top, ps aux, and Hadoop website in=
terface to check the memory.
The values I get from them are different. I also found that the memory alwa=
ys changes periodically.
This is the first thing confused me.

I thought the more files stored in Namenode, the more memory usage in Namen=
ode and Datanode.
I also thought the memory used in Namenode should be larger than the memory=
 used in each Datanode.
However, some results show my ideas are wrong.
For example, I test the memory usage of Namenode with 6000 and 1000 files=
=2E
The "6000" memory is less than "1000" memory from jmap's results.=20
I also found that the memory usage in Datanode is larger than the memory us=
ed in Namenode.

I really don't know how to get the memory usage in Namenode and Datanode.

Can anyone give me some advices?

Thanks,
Tim

--=_f0ab0635288ad0c6e205248c5d4c83f6--