chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saisai Shao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CHUKWA-663) SystemMetrics parse error
Date Thu, 13 Sep 2012 05:41:08 GMT

     [ https://issues.apache.org/jira/browse/CHUKWA-663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Saisai Shao updated CHUKWA-663:
-------------------------------

    Description: 
In some OS environment, Chukwa Agent cannot fetch all the system data, but in the Chukwa Collect,
the format is hard coded, so this will lead to a NullPointException, and this NullPointException
will lead to other Exception.

For example:
In my cluster, swap information can not be retrieved, so this exception will thrown:
java.lang.NullPointerException
        at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.SystemMetrics.parse(SystemMetrics.java:156)
        at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.AbstractProcessor.process(AbstractProcessor.java:82)
        at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.add(HBaseWriter.java:194)
        at org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter.add(SocketTeeWriter.java:252)
        at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.add(PipelineStageWriter.java:40)
        at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.accept(ServletCollector.java:159)
        at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.doPost(ServletCollector.java:208)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
AbstractProcessor class catch this error and saved in a error chunk, this error chunk is also
packaged in KeyValue format and put in HBase, but there is no related HBase column family
 named SystemMetricsInError, so another exception is thrown:
HBaseWriter - org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: DoNotRetryIOException: 1 time, servers with issues: sr116:60020,
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1591)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1367)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:945)
        at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:801)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:784)
        at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
        at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.add(HBaseWriter.java:195)
        at org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter.add(SocketTeeWriter.java:252)
        at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.add(PipelineStageWriter.java:40)
        at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.accept(ServletCollector.java:159)
        at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.doPost(ServletCollector.java:208)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)

There exists a problem:
until I meet parse error, I have already collect some metrics data. But after I throw exception,
the left part of data will not be parsed, so half-integrity data will be put into HBase. If
this half-integrity data do not include "tags:cluster" column, so the cluster name will not
be fetch by HICC, this really is the problem.



  was:'

    
> SystemMetrics parse error
> -------------------------
>
>                 Key: CHUKWA-663
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-663
>             Project: Chukwa
>          Issue Type: Bug
>          Components: Data Collection
>    Affects Versions: 0.6.0
>         Environment: RHEL6.1 Hadoop 1.0.3 HBase 0.94.0
>            Reporter: Saisai Shao
>
> In some OS environment, Chukwa Agent cannot fetch all the system data, but in the Chukwa
Collect, the format is hard coded, so this will lead to a NullPointException, and this NullPointException
will lead to other Exception.
> For example:
> In my cluster, swap information can not be retrieved, so this exception will thrown:
> java.lang.NullPointerException
>         at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.SystemMetrics.parse(SystemMetrics.java:156)
>         at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.AbstractProcessor.process(AbstractProcessor.java:82)
>         at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.add(HBaseWriter.java:194)
>         at org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter.add(SocketTeeWriter.java:252)
>         at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.add(PipelineStageWriter.java:40)
>         at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.accept(ServletCollector.java:159)
>         at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.doPost(ServletCollector.java:208)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:326)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>         at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
> AbstractProcessor class catch this error and saved in a error chunk, this error chunk
is also packaged in KeyValue format and put in HBase, but there is no related HBase column
family  named SystemMetricsInError, so another exception is thrown:
> HBaseWriter - org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
1 action: DoNotRetryIOException: 1 time, servers with issues: sr116:60020,
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1591)
>         at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1367)
>         at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:945)
>         at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:801)
>         at org.apache.hadoop.hbase.client.HTable.put(HTable.java:784)
>         at org.apache.hadoop.hbase.client.HTablePool$PooledHTable.put(HTablePool.java:402)
>         at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.add(HBaseWriter.java:195)
>         at org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter.add(SocketTeeWriter.java:252)
>         at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.add(PipelineStageWriter.java:40)
>         at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.accept(ServletCollector.java:159)
>         at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.doPost(ServletCollector.java:208)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:326)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>         at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
> There exists a problem:
> until I meet parse error, I have already collect some metrics data. But after I throw
exception, the left part of data will not be parsed, so half-integrity data will be put into
HBase. If this half-integrity data do not include "tags:cluster" column, so the cluster name
will not be fetch by HICC, this really is the problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message