incubator-chukwa-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kumar, Suresh" <Suresh.Kum...@emc.com>
Subject RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing request: Read timed out
Date Wed, 19 Sep 2012 20:26:30 GMT
Thanks Saisai, I removed the disabled file, still see the exception.

 

What version of chukwa are you running? What version of HBase are you running?

Looking at the exception in hicc.out, looks like it is a Zookeeper exception which probably

originates from HBase.

 

2012-09-19 14:57:17,139 INFO 2124414527@qtp-1510130526-7-SendThread(localhost:2181) ClientCnxn
- Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x139df9f4b070005,
negotiated timeout = 40000

2012-09-19 14:57:19,142 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK
is null on connection event -- see stack trace for the stack trace when constructor was called
on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING

   

 

Suresh

 

 

From: Shao, Saisai [mailto:saisai.shao@intel.com] 
Sent: Tuesday, September 18, 2012 6:06 PM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing
request: Read timed out

 

Hi Suresh,

I met this exception before.

ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException

        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)

        at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)

        at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)

        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)

        at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)

       at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)

        at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)

        at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)

        at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)

        at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)

I think this is a bug in HICC while reading descriptors from HDFS, at that time I delete this
file (/chukwa/hicc/widgets/disabled) on hdfs and restart HICC, you can try it.

I¡¯m sorry for the other Zookeeper exceptions, also In my environment, HICC only take several
minutes to load.

 

Thanks

Saisai

 

 

From: Kumar, Suresh [mailto:Suresh.Kumar4@emc.com] 
Sent: Wednesday, September 19, 2012 1:45 AM
To: chukwa-user@incubator.apache.org
Subject: RE: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing
request: Read timed out

 

I had checked these before posting, the problem was in the ¡°initial_adaptors¡± file, I had
copied 

¡°add FileTailer FooData /tmp/foo 0¡± from the Chukwa agent documentation, it didn¡¯t work.
I had

to change to  ¡°add filetailer.FileTailingAdaptor FooData /tmp/foo 0¡± to make it work. I
found the

exception in ¡°agent.log¡± and fixed the problem.

 

However, localhost:4080/hicc took over 3 hours to load, the hicc.log had this exception, any
clues?

 

Thanks,

Suresh

 

2012-09-17 18:04:25,698 ERROR 2124414527@qtp-1510130526-7-EventThread ZooKeeperWatcher - ZK
is null on connection event -- see stack trace for the stack trace when constructor was called
on this zkw

java.lang.Exception: ZKW CONSTRUCTOR STACK TRACE FOR DEBUGGING

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:142)

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:126)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1307)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:579)

        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:570)

        at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:187)

        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:195)

        at org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:36)

        at org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:268)

        at org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:198)

        at org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:173)

        at org.apache.hadoop.chukwa.datastore.ChukwaHBaseStore.getClusterNames(ChukwaHBaseStore.java:231)

        at org.apache.hadoop.chukwa.hicc.ClusterConfig.<init>(ClusterConfig.java:63)

        at org.apache.jsp.index_jsp._jspService(org.apache.jsp.index_jsp:99)

        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)

        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)

        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)

        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

        at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)

        at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)

        at org.mortbay.jetty.servlet.DefaultServlet.doGet(DefaultServlet.java:503)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:740)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)

        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)

        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)

        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)

        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)

        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

        at org.mortbay.jetty.Server.handle(Server.java:326)

        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
2012-09-17 18:04:25,700 ERROR 2124414527@qtp-1510130526-7-EventThread ClientCnxn - Error while
calling watcher

java.lang.NullPointerException: ZK is null

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:338)

        at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:275)

        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)

        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)

        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)

        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)

        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

2012-09-17 18:04:28,694 INFO 2124414527@qtp-1510130526-7 RecoverableZooKeeper - The identifier
of this process is 18719@localhost

2012-09-17 18:05:14,141 INFO 1790591505@qtp-1510130526-5 /hicc - jsp: init

2012-09-17 19:45:27,547 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf
is /opt/chukwa/etc/chukwa

2012-09-17 19:45:28,125 INFO 1983406566@qtp-1510130526-10 ChukwaConfiguration - chukwaConf
is /opt/chukwa/etc/chukwa

2012-09-17 19:45:28,421 ERROR 1983406566@qtp-1510130526-10 WidgetBean - java.lang.NullPointerException

        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.<init>(WidgetBean.java:55)

        at org.apache.hadoop.chukwa.datastore.WidgetStore.cacheWidgets(WidgetStore.java:98)

        at org.apache.hadoop.chukwa.datastore.WidgetStore.list(WidgetStore.java:121)

        at org.apache.hadoop.chukwa.rest.bean.WidgetBean.update(WidgetBean.java:158)

        at org.apache.hadoop.chukwa.rest.bean.ColumnBean.update(ColumnBean.java:63)

       at org.apache.hadoop.chukwa.rest.bean.PagesBean.update(PagesBean.java:83)

        at org.apache.hadoop.chukwa.rest.bean.ViewBean.update(ViewBean.java:127)

        at org.apache.hadoop.chukwa.datastore.ViewStore.load(ViewStore.java:92)

        at org.apache.hadoop.chukwa.datastore.ViewStore.<init>(ViewStore.java:61)

        at org.apache.hadoop.chukwa.rest.resource.ViewResource.getView(ViewResource.java:52)

 

 

 

From: ÉÛÈüÈü [mailto:sai.sai.shao@gmail.com] 
Sent: Monday, September 17, 2012 6:28 PM
To: chukwa-user@incubator.apache.org
Subject: Re: agent.log I/O exception (java.net.SocketTimeoutException) caught when processing
request: Read timed out

 

Hi, it seems like a network problem. please:
1. check whether your Chukwa Collector has been started correctly, and bind correct port.
2. maybe port 8081 has been occupied by other process, so please check the collecor.log to
see if there is any exception.
3. check your Chukwa Agent and Collector configure to make sure it is correct.

Thanks
Saisai

2012/9/18 Kumar, Suresh <Suresh.Kumar4@emc.com>

I have the following in my local host :

?u>

hadoop-1.0.3

chukwa-incubating-0.5.0

hbase-0.92.2

?u>

Here are some relevant properties from ?b>chukwa-collector-conf.xml

?u>

<property>

  ?<name>writer.hdfs.filesystem</name>

  ?<value>hdfs://localhost:9000</value>

  ?<description>HDFS to dump to</description>

?</property>

……?u>

<property>

  ?<name>chukwaCollector.http.port</name>

  ?<value>8081</value>

  ?<description>The HTTP port number the collector will listen on</description>

?</property>

?u>

and collectors

?u>

http://localhost:8081

?u>

I see the following in agent.log and http://localhost:4080/hicc is spinning, any help appreciated.

?u>

2012-09-17 16:00:28,698 INFO main Log4JMetricsContext - Log4JMetricsContext.chukwaAgent.period=60

2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_HOME: [/opt/chukwa-incubating-0.5.0]

2012-09-17 16:00:28,910 INFO main ChukwaAgent - Config - CHUKWA_CONF_DIR: [/opt/chukwa/etc/chukwa]

2012-09-17 16:00:29,619 INFO main ChukwaAgent - Config - CHECKPOINT_BASE_NAME: [chukwa_agent_checkpoint]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - checkpointDir: [/opt/chukwa/log]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - CHECKPOINT_INTERVAL_MS: [5000]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - DO_CHECKPOINT_RESTORE: [true]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - STATS_INTERVAL_MS: [10000]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - Config - tags: [cluster="demo"]

2012-09-17 16:00:29,620 INFO main ChukwaAgent - checkpoints are enabled, period is 5000

2012-09-17 16:00:29,620 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/log/chukwa_agent_checkpoint0

2012-09-17 16:00:29,648 INFO main ChukwaAgent - started a new adaptor, id = adaptor_93df4746476c9a4b624f6755b122f9dc
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@30384065]

2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_79203ebcc495e11b747254e660380620
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@212bcd4b]

2012-09-17 16:00:29,651 INFO main ChukwaAgent - started a new adaptor, id = adaptor_e2b39549cac6e966126df286a79ae6ad
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@296f25a7]

2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c79bf882974a14286cffe29d3d4cf0d6
function=[org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics@a13f991]

2012-09-17 16:00:29,667 INFO main ChukwaAgent - started a new adaptor, id = adaptor_c6fd1e5021ad8023477ef0b56a24a29f
function=[org.apache.hadoop.chukwa.datacollection.adaptor.SocketAdaptor@7dc05ffd]

2012-09-17 16:00:29,667 INFO main ChukwaAgent - starting adaptors listed in /opt/chukwa/etc/chukwa/initial_adaptors

2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - AgentControlSocketListerner
ask for port: 9093

2012-09-17 16:00:29,675 INFO main AgentControlSocketListener - socket bound to 9093

2012-09-17 16:00:29,675 INFO main ChukwaAgent - control socket started on port 9093

2012-09-17 16:00:29,713 INFO main log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog

2012-09-17 16:00:29,759 INFO main log - jetty-6.1.26

2012-09-17 16:00:29,821 INFO main log - Started SelectChannelConnector@0.0.0.0:9090

2012-09-17 16:00:29,822 INFO main ChukwaAgent - started Chukwa http agent interface on port
9090

2012-09-17 16:00:29,826 INFO Timer-2 ChukwaAgent - writing checkpoint 1

2012-09-17 16:00:29,827 INFO main ChukwaAgent - local agent started on port 9093

2012-09-17 16:00:29,827 INFO HTTP post thread HttpConnector - HttpConnector started at time:1347912029827

2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("CHUKWA_HOME"):
[/opt/chukwa-incubating-0.5.0/]

2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - Config - System.getenv("chukwaConf"):
[/opt/chukwa/etc/chukwa]

2012-09-17 16:00:29,827 INFO HTTP post thread DataFactory - setting up collectors file: /opt/chukwa/etc/chukwa/collectors

2012-09-17 16:00:29,866 INFO HTTP post thread HttpConnector - using collectors from collectors
file

2012-09-17 16:00:29,926 INFO Timer-3 HttpConnector - # http chunks ACK'ed since last report:
0

2012-09-17 16:00:43,886 INFO HTTP post thread ChukwaHttpSender - collected 1 chunks for post_0

2012-09-17 16:00:43,943 INFO HTTP post thread ChukwaHttpSender - >>>>>>
HTTP post_0 to http://localhost:8081/ length = 248

2012-09-17 16:00:44,037 ERROR HTTP post thread ChukwaHttpSender - Http post exception on http://localhost:8081/:
java.net.ConnectException: Connection refused

2012-09-17 16:00:44,037 WARN HTTP post thread ChukwaHttpSender - No more collectors to try
rolling over to; waiting 20000 ms (144000 retries left)

2012-09-17 16:01:28,711 INFO Thread-20 SocketAdaptor - Caught java.io.EOFException closing
conneciton.

 


Mime
View raw message