incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ratner, Alan S (IS)" <Alan.Rat...@ngc.com>
Subject Chukwa Collector Error (?)
Date Tue, 25 May 2010 19:51:42 GMT
I've installed Chukwa, HICC, Tomcat & Mysql (following the instructions
in http://wiki.apache.org/hadoop/Chukwa_Console_Integration_Guide).
Firefox can open the Tomcat site at http://localhost:8080/ but it is
unable to open http://localhost:8080/hicc giving me just the "waiting
for localhost" message forever.  The collector.log file below indicates
something is amiss.  Although it seems to be complaining that it cannot
connect to HDFS, "bin/hadoop fs -ls" indicates that HDFS is alive and
well.

Any assistance would be appreciated.  Thanks, Alan



ngc@hadoop1:~/chukwa-0.4.0$ bin/start-all.sh
localhost: starting collector, logging to
/tmp/chukwa/log/chukwa-chukwa-collector-hadoop1.out
localhost: 2010-05-25 15:11:41.872::INFO:  Logging to STDERR via
org.mortbay.log.StdErrLog
localhost: 2010-05-25 15:11:41.901::INFO:  jetty-6.1.11
10.64.147.7: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop6.out
10.64.147.8: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop7.out
10.64.147.3: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop2.out
10.64.147.2: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop1.out
10.64.147.4: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop3.out
10.64.147.11: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop10.out
10.64.147.13: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop12.out
10.64.147.16: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop15.out
10.64.147.14: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop13.out
10.64.147.10: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop9.out
10.64.147.20: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop19.out
10.64.147.15: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop14.out
10.64.147.9: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop8.out
10.64.147.30: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop29.out
10.64.147.24: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop23.out
10.64.147.17: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop16.out
10.64.147.25: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop24.out
10.64.147.12: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop11.out
10.64.147.31: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop30.out
10.64.147.27: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop26.out
10.64.147.26: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop25.out
10.64.147.18: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop17.out
10.64.147.33: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop32.out
10.64.147.32: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop31.out
10.64.147.28: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop27.out
10.64.147.22: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop21.out
10.64.147.19: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop18.out
10.64.147.38: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop37.out
10.64.147.23: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop22.out
10.64.147.21: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop20.out
10.64.147.35: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop34.out
10.64.147.29: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop28.out
10.64.147.5: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop4.out
10.64.147.42: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop41.out
10.64.147.39: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop38.out
10.64.147.36: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop35.out
10.64.147.37: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop36.out
10.64.147.41: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop40.out
10.64.147.34: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop33.out
10.64.147.40: starting agent, logging to
/tmp/chukwa/log/chukwa-chukwa-agent-hadoop39.out
starting archive, logging to
/tmp/chukwa/log/chukwa-chukwa-archive-hadoop1.out
starting demux, logging to
/tmp/chukwa/log/chukwa-chukwa-demux-hadoop1.out
starting dp, logging to /tmp/chukwa/log/chukwa-chukwa-dp-hadoop1.out
ngc@hadoop1:~/chukwa-0.4.0$

ngc@hadoop1:/tmp/chukwa/log$ cat agent.log
2010-05-25 15:11:42,626 INFO main ChukwaAgent - Config - CHUKWA_HOME:
[/home/ngc/chukwa-0.4.0/bin/..]
2010-05-25 15:11:42,627 INFO main ChukwaAgent - Config -
CHUKWA_CONF_DIR: [/home/ngc/chukwa-0.4.0/bin/../conf]
2010-05-25 15:11:42,673 INFO main ChukwaAgent - Config -
CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]
2010-05-25 15:11:42,674 INFO main ChukwaAgent - Config - checkpointDir:
[/tmp/chukwa/log]
2010-05-25 15:11:42,674 INFO main ChukwaAgent - Config -
CHECKPOINT_INTERVAL_MS: [5000]
2010-05-25 15:11:42,674 INFO main ChukwaAgent - Config -
DO_CHECKPOINT_RESTORE: [true]
2010-05-25 15:11:42,674 INFO main ChukwaAgent - Config - tags:
[cluster="chukwa"]
2010-05-25 15:11:42,674 INFO main ChukwaAgent - checkpoints are enabled,
period is 5000
2010-05-25 15:11:42,674 INFO main ChukwaAgent - No checkpoints found in
/tmp/chukwa/log
2010-05-25 15:11:42,675 INFO main AgentControlSocketListener -
AgentControlSocketListerner ask for port: 9093
2010-05-25 15:11:42,677 INFO main AgentControlSocketListener - socket
bound to 9093
2010-05-25 15:11:42,677 INFO main ChukwaAgent - control socket started
on port 9093
2010-05-25 15:11:42,683 INFO main ChukwaAgent - local agent started on
port 9093
2010-05-25 15:11:42,683 INFO HTTP post thread HttpConnector -
HttpConnector started at time:1274814702683
2010-05-25 15:11:42,683 INFO HTTP post thread DataFactory - Config -
System.getenv("CHUKWA_HOME"): [/home/ngc/chukwa-0.4.0/bin/../]
2010-05-25 15:11:42,683 INFO HTTP post thread DataFactory - Config -
System.getenv("chukwaConf"): [/home/ngc/chukwa-0.4.0/bin/../conf]
2010-05-25 15:11:42,683 INFO HTTP post thread DataFactory - setting up
collectors file: /home/ngc/chukwa-0.4.0/bin/../conf/collectors
2010-05-25 15:11:42,723 INFO HTTP post thread HttpConnector - using
collectors from collectors file
2010-05-25 15:11:42,780 INFO Timer-1 HttpConnector - # http chunks
ACK'ed since last report: 0
2010-05-25 15:12:42,780 INFO Timer-1 HttpConnector - # http chunks
ACK'ed since last report: 0
...
2010-05-25 15:33:42,783 INFO Timer-1 HttpConnector - # http chunks
ACK'ed since last report: 0
2010-05-25 15:34:42,783 INFO Timer-1 HttpConnector - # http chunks
ACK'ed since last report: 0

ngc@hadoop1:/tmp/chukwa/log$ cat coll*
2010-05-25 15:11:41,246 INFO main ChukwaConfiguration - chukwaConf is
/home/ngc/chukwa-0.4.0/bin/../conf
2010-05-25 15:11:47,415 INFO main root - initing servletCollector
2010-05-25 15:11:47,419 INFO main PipelineStageWriter - using pipelined
writers, pipe length is 2
2010-05-25 15:11:47,424 INFO Thread-6 SocketTeeWriter - listen thread
started
2010-05-25 15:11:47,427 INFO main SeqFileWriter - rotateInterval is
300000
2010-05-25 15:11:47,427 INFO main SeqFileWriter - outputDir is
/chukwa/logs/
2010-05-25 15:11:47,427 INFO main SeqFileWriter - fsname is
hdfs://localhost:9000/
2010-05-25 15:11:47,427 INFO main SeqFileWriter - filesystem type from
core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem
2010-05-25 15:11:48,416 INFO Timer-1 root -
stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
2010-05-25 15:11:48,631 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 0 time(s).
2010-05-25 15:11:49,631 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 1 time(s).
2010-05-25 15:11:50,632 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 2 time(s).
2010-05-25 15:11:51,632 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 3 time(s).
2010-05-25 15:11:52,633 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 4 time(s).
2010-05-25 15:11:53,633 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 5 time(s).
2010-05-25 15:11:54,634 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 6 time(s).
2010-05-25 15:11:55,634 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 7 time(s).
2010-05-25 15:11:56,635 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 8 time(s).
2010-05-25 15:11:57,636 INFO main Client - Retrying connect to server:
localhost/127.0.0.1:9000. Already tried 9 time(s).
2010-05-25 15:11:57,639 ERROR main SeqFileWriter - can't connect to
HDFS, trying default file system instead (likely to be local)
java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on
connection exception: java.net.ConnectException: Connection refused
	at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
	at org.apache.hadoop.ipc.Client.call(Client.java:743)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
	at $Proxy0.getProtocolVersion(Unknown Source)jps

	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
	at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
	at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS
ystem.java:82)
	at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
	at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
	at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
	at
org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFil
eWriter.java:123)
	at
org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(
PipelineStageWriter.java:88)
	at
org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollect
or.init(ServletCollector.java:112)
	at
org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:4
33)
	at
org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
	at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
	at
org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:
616)
	at
org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
	at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513
)
	at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
	at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130
)
	at org.mortbay.jetty.Server.doStart(Server.java:222)
	at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
	at
org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(Col
lectorStub.java:121)
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
	at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja
va:206)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
	at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
	at
org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
	at org.apache.hadoop.ipc.Client.call(Client.java:720)
	... 25 more

ngc@hadoop1:/tmp/chukwa/log$ cat Demux.log 
2010-05-25 15:11:44,976 INFO main ChukwaConfiguration - chukwaConf is
/home/ngc/chukwa-0.4.0/bin/../conf
2010-05-25 15:11:45,279 INFO main DemuxManager - chukwaRootDir:/chukwa/
2010-05-25 15:11:45,279 INFO main DemuxManager -
dataSinkDir:/chukwa/logs/
2010-05-25 15:11:45,279 INFO main DemuxManager -
postProcessDir:/chukwa/postProcess/
2010-05-25 15:11:45,279 INFO main DemuxManager -
archiveRootDir:/chukwa/dataSinkArchives/
2010-05-25 15:11:45,279 INFO main DemuxManager - demuxReducerCount:8
2010-05-25 15:11:45,280 INFO main DemuxManager - Nagios information:
nagiosHost:null, nagiosPort:0, reportingHost:null
2010-05-25 15:11:45,280 WARN main DemuxManager - Alerting is OFF
2010-05-25 15:11:45,288 INFO main DemuxManager - dataSinkDir:
/chukwa/logs/
2010-05-25 15:11:45,288 INFO main DemuxManager - demuxInputDir:
/chukwa/demuxProcessing/mrInput/
2010-05-25 15:11:45,289 INFO main DemuxManager - Demux not ready so
going to sleep ...
2010-05-25 15:12:05,298 INFO main DemuxManager - dataSinkDir:
/chukwa/logs/
2010-05-25 15:12:05,299 INFO main DemuxManager - demuxInputDir:
/chukwa/demuxProcessing/mrInput/
2010-05-25 15:12:05,300 INFO main DemuxManager - Demux not ready so
going to sleep ...
2010-05-25 15:12:25,303 INFO main DemuxManager - dataSinkDir:
/chukwa/logs/
2010-05-25 15:12:25,304 INFO main DemuxManager - demuxInputDir:
/chukwa/demuxProcessing/mrInput/
...
2010-05-25 15:35:45,701 INFO main DemuxManager - Demux not ready so
going to sleep ...
2010-05-25 15:36:05,705 INFO main DemuxManager - dataSinkDir:
/chukwa/logs/
2010-05-25 15:36:05,705 INFO main DemuxManager - demuxInputDir:
/chukwa/demuxProcessing/mrInput/
2010-05-25 15:36:05,706 INFO main DemuxManager - Demux not ready so
going to sleep ...
2010-05-25 15:36:25,709 INFO main DemuxManager - dataSinkDir:
/chukwa/logs/
2010-05-25 15:36:25,709 INFO main DemuxManager - demuxInputDir:
/chukwa/demuxProcessing/mrInput/
2010-05-25 15:36:25,710 INFO main DemuxManager - Demux not ready so
going to sleep ...
2010-05-25 15:36:45,714 INFO main DemuxManager - dataSinkDir:
/chukwa/logs/
2010-05-25 15:36:45,714 INFO main DemuxManager - demuxInputDir:
/chukwa/demuxProcessing/mrInput/
2010-05-25 15:36:45,715 INFO main DemuxManager - Demux not ready so
going to sleep ...
2010-05-25 15:37:05,718 INFO main DemuxManager - dataSinkDir:
/chukwa/logs/
2010-05-25 15:37:05,718 INFO main DemuxManager - demuxInputDir:
/chukwa/demuxProcessing/mrInput/
2010-05-25 15:37:05,719 INFO main DemuxManager - Demux not ready so
going to sleep ...

ngc@hadoop1:/tmp/chukwa/log$ jps
32622 ChukwaAgent
3053 Jps
21274 SecondaryNameNode
26114 Main
31259 QuorumPeerMain
20978 NameNode
25873 Main
407 PostProcessorManager
32722 ChukwaArchiveManager
26355 Main
1277 Bootstrap
330 DemuxManager
18000 org.eclipse.equinox.launcher_1.0.201.R35x_v20090715.jar
31339 ZooKeeperMain
21380 JobTracker

Mime
View raw message