Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A2E68BD99 for ; Fri, 20 Jan 2012 19:57:46 +0000 (UTC) Received: (qmail 58189 invoked by uid 500); 20 Jan 2012 19:57:45 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 58067 invoked by uid 500); 20 Jan 2012 19:57:45 -0000 Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-user@hadoop.apache.org Delivered-To: mailing list hdfs-user@hadoop.apache.org Received: (qmail 58059 invoked by uid 99); 20 Jan 2012 19:57:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Jan 2012 19:57:44 +0000 X-ASF-Spam-Status: No, hits=2.6 required=5.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted nor denied by domain of knoguchi@yahoo-inc.com) Received: from [216.145.54.171] (HELO mrout1.yahoo.com) (216.145.54.171) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Jan 2012 19:57:38 +0000 Received: from SP2-EX07CAS01.ds.corp.yahoo.com (sp2-ex07cas01.corp.sp2.yahoo.com [98.137.59.37]) by mrout1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id q0KJvBlU093927; Fri, 20 Jan 2012 11:57:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yahoo-inc.com; s=cobra; t=1327089431; bh=lUe7mBmrA3YZggk5ApIg4J4vpYWAmTN2citm5LT0vv8=; h=From:To:CC:Date:Subject:Message-ID:In-Reply-To:Content-Type: MIME-Version; b=RC8CPJljxJFUNDYOBD5NCOHozSxrh/nTFpNUz97a23E7l38T3QJw2/QKmjQkWtRp+ a4ZrppiDD62qjQnz9i4WAZEInNDNh8QjjLmCiZxh0XSYWYut1QqN7NCkzTA+SIPux4 iCC4fwdeTpP9kfKIy0GsYWK2JrBPOtgM9jpEZucY= Received: from SP2-EX07VS04.ds.corp.yahoo.com ([98.137.59.33]) by SP2-EX07CAS01.ds.corp.yahoo.com ([98.137.59.37]) with mapi; Fri, 20 Jan 2012 11:57:11 -0800 From: Koji Noguchi To: "hdfs-user@hadoop.apache.org" CC: felix gao Date: Fri, 20 Jan 2012 11:57:04 -0800 Subject: Re: namenode grows overtime Thread-Topic: namenode grows overtime Thread-Index: AQHM1jhQp9oUP1DY3UqqBogztUlnW5YSQmuAgAGsLYCAAQrVmYAAtX7V Message-ID: In-Reply-To: <1542FA4EE20C5048A5C2A3663BED2A6B0F8942FC@szxeml531-mbx.china.huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-Entourage/13.11.0.110726 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CB3F07101D6DFknoguchiyahooinccom_" MIME-Version: 1.0 --_000_CB3F07101D6DFknoguchiyahooinccom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Felix, if no options are specified, then I think what Harsh said makes most= sense. Koji =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Felix, > > Are you noticing this from the NN Web UI or from proper JVM monitoring > tools? Just so you know, the UI had a flaw that has since been fixed: > https://issues.apache.org/jira/browse/HDFS-94 > -- > Harsh J > Customer Ops. Engineer, Cloudera On 1/20/12 1:25 AM, "Uma Maheswara Rao G" wrote: Can you please check in UI, what is the heap usage. Then we can confirm whe= ther java heap is growing or not. top will consider native memory usage also and nio uses directByteBuffers i= nternally. This is good write up from Jonathan https://groups.google.com/group/asynchb= ase/browse_thread/thread/c45bc7ba788b2357 Regards, Uma ________________________________ From: felix gao [gre1600@gmail.com] Sent: Friday, January 20, 2012 6:42 AM To: hdfs-user@hadoop.apache.org Subject: Re: namenode grows overtime Koji, There is no Java options specified other than -Xmx24g, what are some of the= recommended options for namenode? Thanks, Felix On Wed, Jan 18, 2012 at 3:39 PM, Koji Noguchi wrot= e: Hi Felix, Taking jmap =96histo:live would tell you what=92s occupying the heap= . Are you using UseConcMarkSweepGC? If yes, and if you see bunch of java.net.SocksSocketImpl sun.nio.ch.SocketChannelImpl from jmap histo outputs, try passing -XX:-CMSConcurrentMTEnabled . Background: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=3D7113118 Koji On 1/18/12 3:23 PM, "felix gao" > wrote: Hi guys, we are running Hadoop 0.20.2+228, the namenode process's memory grows overt= ime to occupy over 18GB. However, if I restart the namenode, it only occup= ies about 10GB after it is stable. I am wondering if there is anyway to fi= gure out what is going on with namenode that causes it to grow very rapidly= and if there are any tools to make the namenode printout some useful infor= mation on what is holding onto that memory. Thanks, Felix --_000_CB3F07101D6DFknoguchiyahooinccom_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Re: namenode grows overtime Felix, if no options are specified, then I think what Harsh said make= s most sense.

Koji

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Felix,
>
> Are you noticing this from the NN Web UI or from proper JVM monitoring=
> tools? Just so you know, the UI had a flaw that has since been fixed:<= br> > https://issues.apache.org/jira/browse/HDFS-94
> <= font face=3D"Consolas, Courier New, Courier">--
> Harsh J
> Customer Ops. Engineer, Cloudera

On 1/20/12 1:25 AM, "Uma Maheswara Rao G" <maheswara@huawei.com> wrote:

Can you please check in UI,= what is the heap usage. Then we can confirm whether java heap is growing o= r not.

top will consider native memory usage also and nio uses directByteBuffers i= nternally.

This is good write up from Jonathan https://groups.google= .com/group/asynchbase/browse_thread/thread/c45bc7ba788b2357

 

 

Regards,

Uma

<= b>From: felix gao [gre1600@gmail.com]
Sent: Friday, January 20, 2012 6:42 AM
To: hdfs-user@hadoop.apache.= org
Subject: Re: namenode grows overtime

Koji,

There is no Java options specified other than -Xmx24g, what are some of the= recommended options for namenode?

Thanks,

Felix

On Wed, Jan 18, 2012 at 3:39 PM, Koji Noguchi <knoguchi@yahoo-inc.com> wrote:
Hi Felix,

Taking  jmap =96histo:live <pid> would tell you what=92s occupyi= ng the heap.

Are you using  UseConcMarkSweepGC?
If yes, and if you see bunch of
  java.net.SocksSocketImpl
  sun.nio.ch.SocketChannelImpl
from jmap histo outputs, try passing    -XX:-CMSConcurrentMT= Enabled .

Background: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=3D7113118

Koji




On 1/18/12 3:23 PM, "felix gao" <gre1600@gmail.com <http://gre1= 600@gmail.com> > wrote:

Hi guys,

we are running Hadoop 0.20.2+228, the namenode process's memory grows o= vertime to occupy over 18GB.  However, if I restart the namenode, it o= nly occupies about 10GB after it is stable.  I am wondering if there i= s anyway to figure out what is going on with namenode that causes it to gro= w very rapidly and if there are any tools to make the namenode printout som= e useful information on what is holding onto that memory.

Thanks,

Felix



--_000_CB3F07101D6DFknoguchiyahooinccom_--