incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shao, Saisai" <saisai.s...@intel.com>
Subject RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Date Wed, 19 Sep 2012 01:06:29 GMT
Hi Suresh,
I met this exception before.
ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
        at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
        at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
        at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
       at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
        at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
        at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
        at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
        at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)
I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this
file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.
I¡¯m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several
minutes to load.

Thanks
Saisai


From: Kumar, Suresh [mailto:Suresh.Kumar4@emc.com]
Sent: Wednesday, September 19, 2012 1:45 AM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing
request: Read timed out

I had checked these before posting, the problem was in the ¡°initial_adaptors¡± file, I had
copied
¡°add FileTailer FooData /tmp/foo 0¡± from the Chukwa agent documentation, it didn¡¯t work.
I had
to change to  ¡°add filetailer.FileTailingAdaptor FooData /tmp/foo 0¡± to make it work. I
found the
exception in ¡°agent.log¡± and fixed the problem.

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any
clues?

Thanks,
Suresh

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK
is null on connection event -- see stack trace for the stack trace when constructor was called
on this zkw
java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)
        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)
        at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)
        at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)
        at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)
        at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)
        at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)
        at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)
        at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
        at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
        at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
2012-09-17 18:04:25,700 ERROR 2124414527@qtp-1510130526-7-EventThread ClientCnxn - Error while
calling watcher
java.lang.NullPointerException: ZK is null
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:338)
        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:275)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-09-17 18:04:28,694 INFO 2124414527@qtp-1510130526-7 RecoverableZooKeeper - The identifier
of this process is 18719@localhost
2012-09-17 18:05:14,141 INFO 1790591505@qtp-1510130526-5 /hicc - jsp: init
2012-09-17 19:45:27,547 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf
is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,125 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf
is /opt/chukwa/etc/chukwa
2012-09-17 19:45:28,421 ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException
        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)
        at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)
        at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)
        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)
        at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)
       at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)
        at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)
        at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)
        at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)
        at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)



From: ÉÛÈüÈü [mailto:sai.sai.shao@gmail.com]
Sent: Monday, September 17, 2012 6:28 PM
To: chukwa-user@incubator.apache.org<mailto:chukwa-user@incubator.apache.org>
Subject: Re: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing
request: Read timed out

Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind correct port.
2. maybe port 8081 has been occupied by other process, so please check the collecor.log to
see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is correct.

Thanks
Saisai
2012/9/18 Kumar, Suresh <Suresh.Kumar4@emc.com<mailto:Suresh.Kumar4@emc.com>>
I have the following in my local host :
?u>
hadoop-1.0.3
chukwa-incubating-0.5.0
hbase-0.92.2
?u>
Here are some relevant properties from ?b>chukwa-collector-conf.xml
?u>
<property>
  ?<name>writer.hdfs.filesystem</name>
  ?<value>hdfs://localhost:9000</value>
  ?<description>HDFS to dump to</description>
?</property>
……?u>
<property>
  ?<name>chukwaCollector.http.port</name>
  ?<value>8081</value>
  ?<description>The HTTP port number the collector will listen on</description>
?</property>
?u>
and collectors
?u>
http://localhost:8081
?u>
I see the following in agent.log and http://localhost:4080/hicc is spinning, any help appreciated.
?u>
2012-09-17 16:00:28,698 INFO main Log4JMetricsContext - Log4JMetricsContext.chukwaAgent.period=60
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME: [/opt/chukwa-incubating-0.5.0]
2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR: [/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config - CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir: [/opt/chukwa/log]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - CHECKPOINT_INTERVAL_MS: [5000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - DO_CHECKPOINT_RESTORE: [true]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - STATS_INTERVAL_MS: [10000]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags: [cluster="demo"]
2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled, period is 5000
2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/log/chukwa_agent_checkpoint0
2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id = adaptor_93df4746476c9a4b624f6755b122f9dc
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_79203ebcc495e11b747254e660380620
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b]
2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_e2b39549cac6e966126df286a79ae6ad
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c79bf882974a14286cffe29d3d4cf0d6
function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c6fd1e5021ad8023477ef0b56a24a29f
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd]
2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/etc/chukwa/initial_adaptors
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - AgentControlSocketListerner
ask for port: 9093
2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket bound to 9093
2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on port 9093
2012-09-17 16:00:29,713 INFO main log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26
2012-09-17 16:00:29,821 INFO main log - Started SelectChannelConnector@0.0.0.0:9090<http://SelectChannelConnector@0.0.0.0:9090>
2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http agent interface on port
9090
2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1
2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on port 9093
2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector - HttpConnector started at time:1347912029827
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("CHUKWA_HOME"):
[/opt/chukwa-incubating-0.5.0/]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("chukwaConf"):
[/opt/chukwa/etc/chukwa]
2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up collectors file: /opt/chukwa/etc/chukwa/collectors
2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using collectors from collectors
file
2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks ACK'ed since last report:
0
2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender - collected 1 chunks for post_0
2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>>
HTTP post_0 to http://localhost:8081/ length = 248
2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http post exception on http://localhost:8081/:
java.net.ConnectException: Connection refused
2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more collectors to try
rolling over to; waiting 20000 ms (144000 retries left)
2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught java.io.EOFException closing
conneciton.

Mime
View raw message