Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.210.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAAYafM73KEof0NKtrivTvVQbpGEhBHvjHf4nEpUO=BM9-hyJQ@mail.gmail.com>
References: 
 <CAAAYafM73KEof0NKtrivTvVQbpGEhBHvjHf4nEpUO=BM9-hyJQ@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Mon, 2 Jul 2012 15:51:36 +0530
Message-ID: 
 <CAOcnVr3XV1K-osEfbj8gLxCL6+XLAnRKUdF-YthfGkeYMQysvw@mail.gmail.com>
Subject: Re: Namenode hangs on startup
To: hdfs-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Jianhui,

Can you pastebin.com the output of your "jstack <NN PID>" command
after its hung, and pass us the paste link please? It looks to me like
it may have just been merging/saving the image, and that may be slow
but it depends on how long did you have to wait around to see NN
resume and begin properly.

On Mon, Jul 2, 2012 at 2:34 PM, Jianhui Zhang <jhzhang.email@gmail.com> wrote:
> Hi,
>
> Apache Hadoop 0.20.205.
>
> I'm trying to restart NN and it always hangs at the very beginning.
> The only logs I've got are:
>
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG:   host = host/ip
> STARTUP_MSG:   args = []
> STARTUP_MSG:   version = 0.20.205.0
> STARTUP_MSG:   build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-205
> -r 1179940; compiled by 'hortonfo' on Fri Oct  7 06:20:32 UTC 2011
> ************************************************************/
> 2012-07-02 01:33:01,281 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2012-07-02 01:33:01,290 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2012-07-02 01:33:01,292 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2012-07-02 01:33:01,292 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
> system started
> 2012-07-02 01:33:01,434 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> ugi registered.
> 2012-07-02 01:33:01,436 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi
> already exists!
> 2012-07-02 01:33:01,441 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> jvm registered.
> 2012-07-02 01:33:01,441 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> NameNode registered.
> 2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet: VM type
>       = 64-bit
> 2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet: 2% max
> memory = 314.0275 MB
> 2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet:
> capacity      = 2^25 = 33554432 entries
> 2012-07-02 01:33:01,463 INFO org.apache.hadoop.hdfs.util.GSet:
> recommended=33554432, actual=33554432
> 2012-07-02 01:33:01,546 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=owner
> 2012-07-02 01:33:01,546 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> supergroup=supergroup
> 2012-07-02 01:33:01,546 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isPermissionEnabled=true
> 2012-07-02 01:33:01,550 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.block.invalidate.limit=100
> 2012-07-02 01:33:01,550 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
> accessTokenLifetime=0 min(s)
> 2012-07-02 01:33:01,787 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemStateMBean and NameNodeMXBean
> 2012-07-02 01:33:01,802 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2012-07-02 01:33:01,811 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files = 17032
> 2012-07-02 01:33:02,406 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Number of files under
> construction = 0
> 2012-07-02 01:33:02,406 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Image file of size
> 2553316 loaded in 0 seconds.
> 2012-07-02 01:33:02,410 INFO
> org.apache.hadoop.hdfs.server.common.Storage: Edits file
> /apr/hdfs/name/current/edits of size 498 edits # 7 loaded in 0
> seconds.
>
> ====================================
>
> It hangs thereafter.... I wonder if anybody has seen this before?
>
> Some background: I shut down DFS and MR while there were still jobs
> running. Some MR jobs were hanging, so I manually killed the children
> JVMs after the shutdown. Not sure how such actions would affect NN
> startup.
>
> Any help would be appreciated.
>
> Thanks,
> James


-- 
Harsh J