hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jon Lederman <jon2...@gmail.com>
Subject Re: HDFS FS Commands Hanging System
Date Sun, 02 Jan 2011 16:37:09 GMT
Hi,

I followed the example precisely.  It seems to me that the NameNode and DataNode are not communicating.
 I noticed that the log file for my DataNode appears suspiciously short.  I believe it should
try to connect to the NameNode and report such progress.  The log for the DataNode simply
shows:

/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = localhost/127.0.0.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
-r 911707; compiled by 'chrisdo' on F
ri Feb 19 08:07:34 UTC 2010
************************************************************/

Also, the log file for the NameNode indicates 0 racks and 0 DataNodes as indicated in bold:

/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost/127.0.0.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20
-r 911707; compiled by 'chrisdo' on F
ri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-01-02 16:30:34,070 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics
with hostName=NameNode, port=900
0
2011-01-02 16:30:35,093 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up
at: localhost.localdomain/127.0.0.1:90
00
2011-01-02 16:30:35,171 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics
with processName=NameNode, sessi
onId=null
2011-01-02 16:30:35,196 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics:
Initializing NameNodeMeterics using
 context object:org.apache.hadoop.metrics.spi.NullContext
2011-01-02 16:30:37,022 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=root,root
2011-01-02 16:30:37,029 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2011-01-02 16:30:37,032 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
2011-01-02 16:30:37,216 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics:
Initializing FSNamesystemMetric
s using context object:org.apache.hadoop.metrics.spi.NullContext
2011-01-02 16:30:37,242 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStatusMBean
2011-01-02 16:30:37,799 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files
= 1
2011-01-02 16:30:37,882 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files
under construction = 0
2011-01-02 16:30:37,885 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size
94 loaded in 0 seconds.
2011-01-02 16:30:37,891 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /tmp/hadoop-root/dfs/name/current/edits
of
 size 4 edits # 0 loaded in 0 seconds.
2011-01-02 16:30:37,956 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size
94 saved in 0 seconds.
2011-01-02 16:30:38,104 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished
loading FSImage in 1726 msecs
2011-01-02 16:30:38,130 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number
of blocks = 0
2011-01-02 16:30:38,133 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
invalid blocks = 0
2011-01-02 16:30:38,136 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
under-replicated blocks = 0
2011-01-02 16:30:38,139 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
 over-replicated blocks = 0
2011-01-02 16:30:38,144 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode
after 1 secs.
2011-01-02 16:30:38,154 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has
0 racks and 0 datanodes
2011-01-02 16:30:38,159 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks
has 0 blocks
2011-01-02 16:30:41,009 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.
Slf4jLog
2011-01-02 16:30:42,045 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort()
bef
ore open() is -1. Opening the listener on 50070
2011-01-02 16:30:42,060 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned
50070 webServer.getConnectors()
[0].getLocalPort() returned 50070
2011-01-02 16:30:42,062 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2011-01-02 16:30:42,064 INFO org.mortbay.log: jetty-6.1.14

What should I check to see whether there is communication?  Why should the network topology
as reported by the Namenode indicate 0 racks and 0 Datanodes?

Also, I am curious what should be in the masters and slaves files when running in pseudo-distributed
mode.

It seems I need to have both files contain: localhost.  Otherwise, the DataNode and/or NameNode
do not start.

Any help would be greatly appreciated.

Thanks.

-Jon

On Jan 2, 2011, at 3:46 AM, Black, Michael (IS) wrote:

> Did you sert your config and format the namenode as per these instructions?
> 
> http://hadoop.apache.org/common/docs/current/single_node_setup.html
> 
> 
> Michael D. Black
> Senior Scientist
> Advanced Analytics Directorate
> Northrop Grumman Information Systems
> 
> 
> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message