Return-Path: Delivered-To: apmail-hadoop-chukwa-user-archive@minotaur.apache.org Received: (qmail 29114 invoked from network); 25 May 2010 19:52:16 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 25 May 2010 19:52:16 -0000 Received: (qmail 38903 invoked by uid 500); 25 May 2010 19:52:16 -0000 Delivered-To: apmail-hadoop-chukwa-user-archive@hadoop.apache.org Received: (qmail 38887 invoked by uid 500); 25 May 2010 19:52:16 -0000 Mailing-List: contact chukwa-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: chukwa-user@hadoop.apache.org Delivered-To: mailing list chukwa-user@hadoop.apache.org Received: (qmail 38879 invoked by uid 99); 25 May 2010 19:52:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 19:52:15 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Alan.Ratner@ngc.com designates 155.104.240.104 as permitted sender) Received: from [155.104.240.104] (HELO xmrm0101.northgrum.com) (155.104.240.104) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 May 2010 19:52:06 +0000 Received: from xbhm0001.northgrum.com ([155.104.118.90]) by xmrm0101.northgrum.com with InterScan Message Security Suite; Tue, 25 May 2010 15:47:07 -0400 Received: from XBHIL102.northgrum.com ([134.223.165.151]) by xbhm0001.northgrum.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Tue, 25 May 2010 15:51:44 -0400 Received: from XMBIL132.northgrum.com ([134.223.166.142]) by XBHIL102.northgrum.com over TLS secured channel with Microsoft SMTPSVC(6.0.3790.4675); Tue, 25 May 2010 14:51:42 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: Chukwa Collector Error (?) Date: Tue, 25 May 2010 14:51:42 -0500 Message-ID: <68F06E0D1DB9E64A9DB52608086F76B86012ED@XMBIL132.northgrum.com> In-Reply-To: <68F06E0D1DB9E64A9DB52608086F76B858FD39@XMBIL132.northgrum.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Chukwa Collector Error (?) Thread-Index: Acr49LTv9bOlzjLYRX6h/u6AN/mG7gAJdaqrAIffvmAAAu9bwQACdE5gADwgyvA= References: <68F06E0D1DB9E64A9DB52608086F76B858FD1D@XMBIL132.northgrum.com> <68F06E0D1DB9E64A9DB52608086F76B858FD39@XMBIL132.northgrum.com> From: "Ratner, Alan S (IS)" To: X-OriginalArrivalTime: 25 May 2010 19:51:42.0854 (UTC) FILETIME=[B2EFAA60:01CAFC43] X-Virus-Checked: Checked by ClamAV on apache.org I've installed Chukwa, HICC, Tomcat & Mysql (following the instructions in http://wiki.apache.org/hadoop/Chukwa_Console_Integration_Guide). Firefox can open the Tomcat site at http://localhost:8080/ but it is unable to open http://localhost:8080/hicc giving me just the "waiting for localhost" message forever. The collector.log file below indicates something is amiss. Although it seems to be complaining that it cannot connect to HDFS, "bin/hadoop fs -ls" indicates that HDFS is alive and well. Any assistance would be appreciated. Thanks, Alan ngc@hadoop1:~/chukwa-0.4.0$ bin/start-all.sh localhost: starting collector, logging to /tmp/chukwa/log/chukwa-chukwa-collector-hadoop1.out localhost: 2010-05-25 15:11:41.872::INFO: Logging to STDERR via org.mortbay.log.StdErrLog localhost: 2010-05-25 15:11:41.901::INFO: jetty-6.1.11 10.64.147.7: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop6.out 10.64.147.8: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop7.out 10.64.147.3: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop2.out 10.64.147.2: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop1.out 10.64.147.4: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop3.out 10.64.147.11: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop10.out 10.64.147.13: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop12.out 10.64.147.16: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop15.out 10.64.147.14: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop13.out 10.64.147.10: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop9.out 10.64.147.20: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop19.out 10.64.147.15: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop14.out 10.64.147.9: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop8.out 10.64.147.30: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop29.out 10.64.147.24: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop23.out 10.64.147.17: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop16.out 10.64.147.25: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop24.out 10.64.147.12: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop11.out 10.64.147.31: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop30.out 10.64.147.27: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop26.out 10.64.147.26: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop25.out 10.64.147.18: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop17.out 10.64.147.33: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop32.out 10.64.147.32: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop31.out 10.64.147.28: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop27.out 10.64.147.22: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop21.out 10.64.147.19: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop18.out 10.64.147.38: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop37.out 10.64.147.23: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop22.out 10.64.147.21: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop20.out 10.64.147.35: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop34.out 10.64.147.29: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop28.out 10.64.147.5: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop4.out 10.64.147.42: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop41.out 10.64.147.39: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop38.out 10.64.147.36: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop35.out 10.64.147.37: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop36.out 10.64.147.41: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop40.out 10.64.147.34: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop33.out 10.64.147.40: starting agent, logging to /tmp/chukwa/log/chukwa-chukwa-agent-hadoop39.out starting archive, logging to /tmp/chukwa/log/chukwa-chukwa-archive-hadoop1.out starting demux, logging to /tmp/chukwa/log/chukwa-chukwa-demux-hadoop1.out starting dp, logging to /tmp/chukwa/log/chukwa-chukwa-dp-hadoop1.out ngc@hadoop1:~/chukwa-0.4.0$ ngc@hadoop1:/tmp/chukwa/log$ cat agent.log 2010-05-25 15:11:42,626 INFO main ChukwaAgent - Config - CHUKWA_HOME: [/home/ngc/chukwa-0.4.0/bin/..] 2010-05-25 15:11:42,627 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR: [/home/ngc/chukwa-0.4.0/bin/../conf] 2010-05-25 15:11:42,673 INFO main ChukwaAgent - Config - CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint] 2010-05-25 15:11:42,674 INFO main ChukwaAgent - Config - checkpointDir: [/tmp/chukwa/log] 2010-05-25 15:11:42,674 INFO main ChukwaAgent - Config - CHECKPOINT_INTERVAL_MS: [5000] 2010-05-25 15:11:42,674 INFO main ChukwaAgent - Config - DO_CHECKPOINT_RESTORE: [true] 2010-05-25 15:11:42,674 INFO main ChukwaAgent - Config - tags: [cluster=3D"chukwa"] 2010-05-25 15:11:42,674 INFO main ChukwaAgent - checkpoints are enabled, period is 5000 2010-05-25 15:11:42,674 INFO main ChukwaAgent - No checkpoints found in /tmp/chukwa/log 2010-05-25 15:11:42,675 INFO main AgentControlSocketListener - AgentControlSocketListerner ask for port: 9093 2010-05-25 15:11:42,677 INFO main AgentControlSocketListener - socket bound to 9093 2010-05-25 15:11:42,677 INFO main ChukwaAgent - control socket started on port 9093 2010-05-25 15:11:42,683 INFO main ChukwaAgent - local agent started on port 9093 2010-05-25 15:11:42,683 INFO HTTP post thread HttpConnector - HttpConnector started at time:1274814702683 2010-05-25 15:11:42,683 INFO HTTP post thread DataFactory - Config - System.getenv("CHUKWA_HOME"): [/home/ngc/chukwa-0.4.0/bin/../] 2010-05-25 15:11:42,683 INFO HTTP post thread DataFactory - Config - System.getenv("chukwaConf"): [/home/ngc/chukwa-0.4.0/bin/../conf] 2010-05-25 15:11:42,683 INFO HTTP post thread DataFactory - setting up collectors file: /home/ngc/chukwa-0.4.0/bin/../conf/collectors 2010-05-25 15:11:42,723 INFO HTTP post thread HttpConnector - using collectors from collectors file 2010-05-25 15:11:42,780 INFO Timer-1 HttpConnector - # http chunks ACK'ed since last report: 0 2010-05-25 15:12:42,780 INFO Timer-1 HttpConnector - # http chunks ACK'ed since last report: 0 ... 2010-05-25 15:33:42,783 INFO Timer-1 HttpConnector - # http chunks ACK'ed since last report: 0 2010-05-25 15:34:42,783 INFO Timer-1 HttpConnector - # http chunks ACK'ed since last report: 0 ngc@hadoop1:/tmp/chukwa/log$ cat coll* 2010-05-25 15:11:41,246 INFO main ChukwaConfiguration - chukwaConf is /home/ngc/chukwa-0.4.0/bin/../conf 2010-05-25 15:11:47,415 INFO main root - initing servletCollector 2010-05-25 15:11:47,419 INFO main PipelineStageWriter - using pipelined writers, pipe length is 2 2010-05-25 15:11:47,424 INFO Thread-6 SocketTeeWriter - listen thread started 2010-05-25 15:11:47,427 INFO main SeqFileWriter - rotateInterval is 300000 2010-05-25 15:11:47,427 INFO main SeqFileWriter - outputDir is /chukwa/logs/ 2010-05-25 15:11:47,427 INFO main SeqFileWriter - fsname is hdfs://localhost:9000/ 2010-05-25 15:11:47,427 INFO main SeqFileWriter - filesystem type from core-default.xml is org.apache.hadoop.hdfs.DistributedFileSystem 2010-05-25 15:11:48,416 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0 2010-05-25 15:11:48,631 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 0 time(s). 2010-05-25 15:11:49,631 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 1 time(s). 2010-05-25 15:11:50,632 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 2 time(s). 2010-05-25 15:11:51,632 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 3 time(s). 2010-05-25 15:11:52,633 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 4 time(s). 2010-05-25 15:11:53,633 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 5 time(s). 2010-05-25 15:11:54,634 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 6 time(s). 2010-05-25 15:11:55,634 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 7 time(s). 2010-05-25 15:11:56,635 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 8 time(s). 2010-05-25 15:11:57,636 INFO main Client - Retrying connect to server: localhost/127.0.0.1:9000. Already tried 9 time(s). 2010-05-25 15:11:57,639 ERROR main SeqFileWriter - can't connect to HDFS, trying default file system instead (likely to be local) java.net.ConnectException: Call to localhost/127.0.0.1:9000 failed on connection exception: java.net.ConnectException: Connection refused at org.apache.hadoop.ipc.Client.wrapException(Client.java:767) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy0.getProtocolVersion(Unknown Source)jps at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:207) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:170) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS ystem.java:82) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter.init(SeqFil eWriter.java:123) at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init( PipelineStageWriter.java:88) at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollect or.init(ServletCollector.java:112) at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:4 33) at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39) at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java: 616) at org.mortbay.jetty.servlet.Context.startContext(Context.java:140) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513 ) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130 ) at org.mortbay.jetty.Server.doStart(Server.java:222) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39) at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(Col lectorStub.java:121) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.ja va:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304) at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176) at org.apache.hadoop.ipc.Client.getConnection(Client.java:860) at org.apache.hadoop.ipc.Client.call(Client.java:720) ... 25 more ngc@hadoop1:/tmp/chukwa/log$ cat Demux.log=20 2010-05-25 15:11:44,976 INFO main ChukwaConfiguration - chukwaConf is /home/ngc/chukwa-0.4.0/bin/../conf 2010-05-25 15:11:45,279 INFO main DemuxManager - chukwaRootDir:/chukwa/ 2010-05-25 15:11:45,279 INFO main DemuxManager - dataSinkDir:/chukwa/logs/ 2010-05-25 15:11:45,279 INFO main DemuxManager - postProcessDir:/chukwa/postProcess/ 2010-05-25 15:11:45,279 INFO main DemuxManager - archiveRootDir:/chukwa/dataSinkArchives/ 2010-05-25 15:11:45,279 INFO main DemuxManager - demuxReducerCount:8 2010-05-25 15:11:45,280 INFO main DemuxManager - Nagios information: nagiosHost:null, nagiosPort:0, reportingHost:null 2010-05-25 15:11:45,280 WARN main DemuxManager - Alerting is OFF 2010-05-25 15:11:45,288 INFO main DemuxManager - dataSinkDir: /chukwa/logs/ 2010-05-25 15:11:45,288 INFO main DemuxManager - demuxInputDir: /chukwa/demuxProcessing/mrInput/ 2010-05-25 15:11:45,289 INFO main DemuxManager - Demux not ready so going to sleep ... 2010-05-25 15:12:05,298 INFO main DemuxManager - dataSinkDir: /chukwa/logs/ 2010-05-25 15:12:05,299 INFO main DemuxManager - demuxInputDir: /chukwa/demuxProcessing/mrInput/ 2010-05-25 15:12:05,300 INFO main DemuxManager - Demux not ready so going to sleep ... 2010-05-25 15:12:25,303 INFO main DemuxManager - dataSinkDir: /chukwa/logs/ 2010-05-25 15:12:25,304 INFO main DemuxManager - demuxInputDir: /chukwa/demuxProcessing/mrInput/ ... 2010-05-25 15:35:45,701 INFO main DemuxManager - Demux not ready so going to sleep ... 2010-05-25 15:36:05,705 INFO main DemuxManager - dataSinkDir: /chukwa/logs/ 2010-05-25 15:36:05,705 INFO main DemuxManager - demuxInputDir: /chukwa/demuxProcessing/mrInput/ 2010-05-25 15:36:05,706 INFO main DemuxManager - Demux not ready so going to sleep ... 2010-05-25 15:36:25,709 INFO main DemuxManager - dataSinkDir: /chukwa/logs/ 2010-05-25 15:36:25,709 INFO main DemuxManager - demuxInputDir: /chukwa/demuxProcessing/mrInput/ 2010-05-25 15:36:25,710 INFO main DemuxManager - Demux not ready so going to sleep ... 2010-05-25 15:36:45,714 INFO main DemuxManager - dataSinkDir: /chukwa/logs/ 2010-05-25 15:36:45,714 INFO main DemuxManager - demuxInputDir: /chukwa/demuxProcessing/mrInput/ 2010-05-25 15:36:45,715 INFO main DemuxManager - Demux not ready so going to sleep ... 2010-05-25 15:37:05,718 INFO main DemuxManager - dataSinkDir: /chukwa/logs/ 2010-05-25 15:37:05,718 INFO main DemuxManager - demuxInputDir: /chukwa/demuxProcessing/mrInput/ 2010-05-25 15:37:05,719 INFO main DemuxManager - Demux not ready so going to sleep ... ngc@hadoop1:/tmp/chukwa/log$ jps 32622 ChukwaAgent 3053 Jps 21274 SecondaryNameNode 26114 Main 31259 QuorumPeerMain 20978 NameNode 25873 Main 407 PostProcessorManager 32722 ChukwaArchiveManager 26355 Main 1277 Bootstrap 330 DemuxManager 18000 org.eclipse.equinox.launcher_1.0.201.R35x_v20090715.jar 31339 ZooKeeperMain 21380 JobTracker