hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted <r6squee...@gmail.com>
Subject pauses during startup (maybe network related?)
Date Thu, 23 May 2013 09:49:47 GMT
Hi I'm running hadoop on my local laptop for development and
everything "works" but there's some annoying pauses during the startup
which causes the entire hadoop startup process to take up to 4 minutes
and I'm wondering what it is and if I can do anything about it.

I'm running everything on 1 machines, on fedora linux, hadoop-1.1.2,
oracle jkd1.7.0_17, the machine is a dual core i5, and I have 8gb of
ram and an SSD so it shouldn't be slow.

When the system pauses, there is no cpu usage, no disk usage and no
network usage (although I suspect it's waiting for the network to
resolve or return something).

Here's some snippets from the namenode logs during startup where you
can see it just pauses for around 30 seconds or more with out errors
or anything :

2013-05-23 19:26:37,660 INFO
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
2013-05-23 19:26:37,676 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2013-05-23 19:27:54,144 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-05-23 19:27:54,144 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system started
2013-05-23 19:27:54,341 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: The
dfs.support.append option is in your configuration, however append is
not supported. This configuration option is no longer required to
enable sync.
2013-05-23 19:27:54,341 INFO
isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
accessTokenLifetime=0 min(s)
2013-05-23 19:28:19,918 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
FSNamesystemStateMBean and NameNodeMXBean
2013-05-23 19:28:19,937 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2013-05-23 19:28:26,801 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 28 on 9000: starting
2013-05-23 19:28:26,833 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 31 on 9000: starting
2013-05-23 19:30:10,644 INFO org.apache.hadoop.hdfs.StateChange:
BLOCK* NameSystem.registerDatanode: node registration from storage DS-651015167-
2013-05-23 19:30:10,650 INFO org.apache.hadoop.net.NetworkTopology:
Adding a new node: /default-rack/

I already start the system with : export
I only allocate : export HADOOP_HEAPSIZE=512 (but it's an empty hadoop
system, maybe just 1 or 2 test files less than 100k, and there's no
CPU usage so it doesn't look like it's GC thrashing)

I should mention again, there's no errors and the system runs fine and
relatively speedy once started (considering it's on my laptop).

Does anyone know what's causing these pauses? (and how I can get rid of them)

View raw message